Product (concept)

A product is the central concept of the processor. A product is a dataset (much like netCDF or xarray defines it), associated with some rules. These rules describe what prequisites must be met to sensibly create such a product and they state how a product is made out of these prequisites.

Products have properties and parts. Properties are metadata like date or integration time and are stored in a way that they can be efficiently searched for. Parts are the actual data and much like a DataArray in xarray. The content of the parts is not efficiently searcheable, because it tends to be large and is typically not even stored but recalculated on every request.

Products mostly split up into imported and derived products.

imported product

imported products are products, created from existing data (like measurement data or calibration data).

Their prequisites are thus a description of how the input files look like and can be used to scan a folder for existing data.

Their creation rule states how to open the data files and how to convert the raw data representation into a dataset-like appearence. Imported products are illustrated by a folder symbol:

digraph imported_product { "imported product" [shape="folder"] }

derived product

derived products are products which can be calculated from (at least one) other product(s) (like calibrated data or averaged dark data). The other products are called components.

Their prequisites state how the components must look like and which relations must hold between these components (e.g. a calibration file must be valid in the period where the raw data has been measured and the optics must match as well). These rules can be used to scan the collection of all existing products for possibilities to create new products. This can of course be applied recursively.

The creation rules subsequently state how to calculate the data out of the components. Derived products are illustrated by a rectangular symbol with incoming arrows. The arrows are originated in the component products:

digraph derived_product { "imported product" [shape="folder"] "derived product" [shape="rect"] "imported product" -> "derived product"; }