spacetelescope / jwst

Running OutlierDetectionStep as part of a pipeline (as is done in calwebb_image3) prevents use of in_memory option.

Near the start of process for calwebb_image3, the input association file is opened with datamodels.open:

jwst/jwst/pipeline/calwebb_image3.py

Line 71 in 613383f

with datamodels.open(input_data, asn_exptypes=asn_exptypes) as input_models:

The save_open (in_memory) option is not provided at this point leading to opening of the association (as a ModelContainer) triggering the opening of every file listed in the association. This can be confirmed by opening any association with the same arguments provided above (importantly with save_open set as the default True).

This opened association/ModelContainer is later passed to OutlierDetectionStep:

jwst/jwst/pipeline/calwebb_image3.py

Line 92 in 613383f

input_models = self.outlier_detection(input_models)

where the in_memory argument is used:

jwst/jwst/outlier_detection/outlier_detection_step.py

Line 79 in 613383f

with datamodels.open(input_data, save_open=self.in_memory) as input_models:

but has no effect as the input to datamodels.open is an already opened ModelContainer which does not use the save_open argument:

jwst/jwst/datamodels/container.py

Line 186 in 613383f

elif isinstance(init, self.__class__):

and instead creates references to the already opened models.

Pipeline association handling prevents `in_memory` option for `OutlierDetectionStep`