Giters

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope

Home Page:https://jwst-pipeline.readthedocs.io/en/latest/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Wrapping detector1 with multiprocessing results in error (M1/M2 issue only?)

stscijgbot-jp opened this issue · comments

commented

Issue JP-3586 was created on JIRA by Tyler Pauly:

From spacetelescope/stcal#247 , a user reports that running detector1 in a multiprocessing environment results in errors. The use case seems to be a decision on which of the following works better:

  1. Running N files through detector1 serially, while setting num_cores > 1 for jump and ramp_fit.
  2. Running N files through detector1 in parallel using mp, while setting num_cores == 1 for jump and ramp_fit.

Currently, 1) is functional but perhaps slower than 2), when many files are involved and many threads/cores are available. But 2) appears to be incompatible with the current pipeline, likely due to a multiprocessing call inside a parent multiprocessing call.

commented

This may be related to recent changes to the multiprocessing start method which a user reported may impact other code:
#8306
Also see a PR that addresses start method issue:
#8343

commented

Comment by Ned Molter on JIRA:

The user's script runs just fine on my Mac with M2 processors using the latest pipeline version, so I believe this has been solved by the PR 8343 linked above and can be closed.  Brett Graham I wrote a comment on the stcal GitHub issue, but I don't think I have permission to close the issue myself.  If you are happy with what I wrote there, and you think the test I ran is enough evidence that this is no longer an issue, you can feel free to close that issue.

commented

Comment by Ned Molter on JIRA:

Already fixed by the PR mentioned above, as well as the documentation updates in #8408