CL example 20 sq degree run on NERSC
cavestruz opened this issue · comments
The example notebook, txpipe_cluster_background_selection.ipynb contains a section with instructions for running TXPipe to select cluster background sources on the 20 square degree CosmoDC2 data + mock noise dataset. There are yaml files and a .sub script to submit a job to IN2P3. But, there is no parallel set of files and instructions for running the same job on NERSC.
Goals:
- Look into example jobs to run on NERSC
- Create an analogous
.subscript to runexamples/cosmodc2/20deg2-in2p3.sub - Check for any differences in necessary yaml files, create new (decide whether to put into subdirectories or same directory)
- Modify
txpipe_cluster_background_selection.ipynbto include NERSC version of instructions. - Test
I can put together a submission script for this, it should be easy (much easier than in2p3!).
One thing to note - on NERSC I've always used the shifter environment for TXPipe rather than conda, but that makes it take a bit longer to install new versions of things. We have to
- modify the Dockerfile here to add stuff
- launch the action to build it
- pull it at nersc with
shifterimg pull ghcr.io/lsstdesc/txpipe-dev
I have started this now.
It should be possible to do a conda environment too but I haven't tried that recently.
I've added a submission script and pipeline file to the master branch.
Please do move them around or otherwise do what you like with them!
Ah, okay - cool, thank you so much! I'll test this out.
Quick note - the conda instructions do in fact work.
Closed with PR #313