Handle flat covariance in error propagation

[stscijgbot-jp]

Issue JP-3575 was created on JIRA by Melanie Clarke:

Currently, the pipeline propagates variance through the resampling step by adding read noise, Poisson noise, and flat variance components from each exposure.  Read noise and Poisson noise for each exposure are statistically independent for each exposure so they have no covariance. 

Flat variance is not generally statistically independent for each exposure: most commonly, each exposure is reduced with the same flat field.  This introduces a covariance term between the exposures that should be accounted for in error propagation when the exposures are combined.

For NIRSpec, the flat errors include the systematic flux calibration error which is not small compared to the statistical errors for the exposure, and should not average down in the same way the statistical errors do when the exposures are combined.  The covariance term is likely significant for NIRSpec at least, and may impact error propagation for other instruments as well.

[stscijgbot-jp]

Comment by Charles Proffitt on JIRA:

Currently the error vectors in the FFLAT files are derived by comparing the extracted level 3 spectrum to the adopted model.  The median fractional error in these vectors, at least for the recent MOS recalibration, is ~ about 0.5%, with spikes up to a few percent, but remember that already includes the merger of typically 3 dither positions (or more for the IFU) and so is perhaps an underestimate of any fixed pattern noise in an individual exposure.  So taking repeated exposures doesn't really reduce this. So for now this probably represents an upper limit to the achievable S/N regardless of how many positions are dithered over or observations combined.  Even if we're converging on the proper net count rate to much better precision, the FFLATs won't reflect this and so the error will remain.