Data pipeline products and their description

[MateuszKM]

The readthedocs description of the pipeline products needs a bit of work. This is somewhat related to (retired) issues #38 and #39

1. Divide the table into engineering and science data products. Former being used for debugging the data.
2. Include sky spectrum (if subtracted) and object cube in the pipeline output
3. Include a description of all fits file extensions in a file, including physical units and, in the case of flags/masks, an explanation of values.
4. Both 2d and cube image data is given as "/pixel", even though the nature of the pixels changes between the two types. Are fluxes really per pixel in the final product rather than physical units?

[Max Brodheim]

Extending this to propose modifying nomenclature to better reflect data products:

• Rename uncertainty to variance
• Perhaps using cube instead of icube since the distinction is less critical in this pipeline
• Instead of using single letter suffixes, use full words

[Don Neill]

If you use the ftools program fv to examine the pipeline output files, this is what you discover.
Each file has four extensions: 0 through 3 or ['Primary', 'MASK', 'UNCERT', 'FLAGS']. Each
of these extensions has a separate header and they give the following units:

Primary: (whatever BUNIT is set to)
MASK: (None)
UNCERT: StdDevUncertainty
FLAGS: (None)
These are defined by the CCDData class which comes from astropy.nddata.

For the 'int' files: 'int', 'intf', 'intd', 'intk' and 'sky' the bunit says: 'electron' / Pixel units

For the 'icube', 'icubew', and 'icubed' files, the bunit says: 'electron' / Pixel units

For the 'icubes' file, the bunit says: '1e+16 erg / (Angstrom cm2 s)' / Pixel units

So, I recommend that any place where the uncertainty is referred to as 'variance' be changed to
refer to it as 'StdDev'.

Answering the question as to if the flux calibrated units should be per pixel or per square arcsecond is
not trivial. It involves re-learning the calibration procedure in exquisite detail. I'll attempt to do this
(for the umpteenth time) and come to a conclusion.

[Don Neill]

This also explains why the flux calibrated 'Uncertainty' is different between the two pipelines. The KCWI_DRP primitive, FluxCalibrate.py treats the uncertainty image like a variance and multiplies is by cal ** 2, when it should be multiplied by
just the cal since it is, indeed, StdDev. I will submit a pull request for that calculation in FluxCalibrate.py.

[Don Neill]

So the flux units are in erg s-1 cm-2 Ang-1, according to the headers of the flux standard files. Our procedure attempts to
gather all the light for the standard by summing over a large aperture. Since these are stars having no real extent, there is no areal sky unit in the calibration. Not exactly sure how to convert to an areal unit (arcsec-1), but it begs the question: what is the calibration for extended sources?

[Don Neill]

I also want to record that we need to decide how to handle the sky products before we rename 'icube'. Specifically, if we decide to make a separate cube for sky, it should be called 'scube' and 'icube' maybe should remain. Alternately, we could add the sky cube as a separate extension, although, it should have it's own uncertainty, mask, and flag extensions, so I'm not sure that is a good idea.

Also, current nomenclature dictates that the object cube 'ocube' is the un-sky-subtracted frame (specifically in a nod-and-shuffle observation), so this would preclude using 'ocube' for an ordinary, sky-subtracted science cube.

All this is to say that we need to think of all facets before we change any nomenclature.