The prefect-slurm infrastructure allows it to run prefect workflows on HPC infrastructure mananged by SLURM.
Create an infrastructure block
import prefect_slurm
from prefect_slurm import SlurmJob
def create_slurm_job() -> SlurmJob:
infra = SlurmJob(
host="<DNS name of the HPC login node>",
username="<USERNAME>",
slurm_kwargs={
"jobname": "prefect",
"partition": "small",
"nodes": "1",
"ntasks-per-node": "72"
},
)
return infra
if __name__ == "__main__":
create_slurm_job().save("hpc", overwrite=True)
Configure a deployment to run on slurm
prefect deployment build my_flow.py:hpc_job -ib slurmjob/hpc --name hpc/job
Install prefect-shell
with pip
:
pip install -U 'prefect-slurm @ git+https://github.com/kramergroup/prefect-slurm'
Requires an installation of Python 3.7+.
We recommend using a Python virtual environment manager such as pipenv, conda or virtualenv.
This infrastructure block is designed to work with Prefect 2. For more information about how to use Prefect, please refer to the Prefect documentation.
If you encounter any bugs while using prefect-slurm
, feel free to open an issue in the prefect-slurm repository.
Feel free to star or watch prefect-slurm
for updates too!
If you'd like to help contribute to fix an issue or add a feature to prefect-slurm
, please propose changes through a pull request from a fork of the repository.
Here are the steps:
- Fork the repository
- Clone the forked repository
- Install the repository and its dependencies:
pip install -e ".[dev]"
- Make desired changes
- Add tests
- Insert an entry to CHANGELOG.md
- Install
pre-commit
to perform quality checks prior to commit:
pre-commit install
git commit
,git push
, and create a pull request