slurm

There are 10 repositories under slurm topic.

stas00 / ml-engineering
Machine Learning Engineering Open Book
ai inference large-language-models llm machine-learning machine-learning-engineering mlops pytorch scalability slurm training transformers
Language:Python 15640
SchedMD / slurm
Slurm: A Highly Scalable Workload Manager
slurm slurm-job-scheduler slurm-workload-manager
Language:C 3416
nextflow
nextflow-io / nextflow
A DSL for data-driven computational pipelines
bioinformatics workflow-engine pipeline pipeline-framework nextflow cloud groovy sge slurm aws docker singularity hpc singularity-containers reproducible-science reproducible-research dataflow hello
Language:Groovy 3202
dstackai / dstack
dstack is an open-source control plane for running development, training, and inference jobs on GPUs—across hyperscalers, neoclouds, or on-prem.
amd cloud containers docker fine-tuning gpu inference k8s kubernetes llms machine-learning nvidia orchestration python slurm training
Language:Python 1951
facebookincubator / submitit
Python 3.8+ toolbox for submitting jobs to Slurm
slurm python clusters
Language:Python 1539
DataBiosphere / toil
A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.
aws common-workflow-language cwl gridengine kubernetes mesos pipeline python slurm wdl workflow workflow-description-language
Language:Python 920
PySlurm / pyslurm
Python Interface to Slurm
slurm cython python hpc cluster
Language:Cython 548
LambdaLabsML / distributed-training-guide
Best practices & guides on how to write distributed pytorch training code
cuda deepspeed distributed-training gpu gpu-cluster kuberentes nccl pytorch slurm cluster fsdp lambdalabs mpi sharding
Language:Python 530
rackslab / Slurm-web
Open source web interface for Slurm HPC & AI clusters
dashboard hpc slurm webui ai
Language:Python 505
pipefunc
pipefunc / pipefunc
Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪
pipeline-framework pipelines reproducible-research dag hpc parallel-computing slurm workflow-engine
Language:Python 427
giovtorres / slurm-docker-cluster
A Slurm cluster using docker-compose
hpc slurm slurm-cluster docker-compose
Language:Shell 409
meta-pytorch / torchx
TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.
airflow aws-batch components deep-learning distributed-training kubernetes machine-learning pipelines python pytorch ray slurm
Language:Python 399
justanhduc / task-spooler
A scheduler for GPU/CPU tasks
slurm slurm-job slurm-job-scheduler task-spooler job-scheduler cpp linux gpu-support debian c makefile
Language:C 361
elasticluster / elasticluster
Create clusters of VMs on the cloud and configure them with Ansible.
python cluster slurm gridengine hadoop spark ec2 cloud gcp azure clustering hpc ansible
Language:Python 338
soperator
nebius / soperator
Run Slurm in Kubernetes
high-performance-computing hpc k8s k8s-operator kubernetes machine-learning ml model-training slurm
Language:Go 311
omnia
dell / omnia
An open-source toolkit for deploying and managing high performance clusters for HPC, AI, and data analytics workloads.
k8s-cluster kubernetes slurm-cluster slurm dell-emc dellemc ansible ansible-playbooks hpc-clusters hpc stateless
Language:YAML 280
Azure / batch-shipyard
Simplify HPC and Batch workloads on Azure
azure-batch docker hpc mpi gpu infiniband rdma azure batch-processing nfs glusterfs azure-functions containers singularity windows-containers serverless slurm
Language:Python 276
zhenrong-wang / hpc-now
A Cross-Platform, Multi-Cloud High-Performance Computing Platform
aliyun aws azure baiduyun c cloud google-cloud hpc huaweicloud scripts tencent-cloud terraform cluster linux slurm opentofu devops
Language:C 271
vpenso / prometheus-slurm-exporter
Prometheus exporter for performance metrics from Slurm.
prometheus-exporter slurm
Language:Go 269
kabouzeid / turm
TUI for the Slurm Workload Manager
slurm rust terminal tui tui-rs
Language:Rust 222
TUM-DAML / seml
SEML: Slurm Experiment Management Library
utility slurm slurm-workload-manager experiment-tracking hyperparameter-optimization experiment-manager orchestration
Language:Python 185
mlr-org / batchtools
Tools for computation on batch systems
batchexperiments batchjobs cran docker-swarm high-performance-computing hpc hpc-clusters lsf openlava parallel-computing r reproducibility sge slurm torque
Language:R 183
SlinkyProject / slurm-operator
Run Slurm on Kubernetes. A Slinky project.
helm-charts high-performance-computing high-throughput-computing hpc k8s-operator kubernetes schedmd slinky slurm slurm-workload-manager
Language:Go 182
jdblischak / smk-simple-slurm
A simple Snakemake profile for Slurm without --cluster-config
bioinformatics snakemake slurm snakemake-profile
Language:Shell 164
foamlib
gerlero / foamlib
✨ A modern Python package for interacting with OpenFOAM
asyncio openfoam parsing python cases dictionaries fields files slurm joss numpy
Language:Python 159
mschubert / clustermq
R package to send function calls as jobs on LSF, SGE, Slurm, PBS/Torque, or each via SSH
cluster high-performance-computing lsf r-package sge slurm ssh
Language:R 153
gdikov / hypertunity
A toolset for black-box hyperparameter optimisation.
hyperparameter-optimization bayesian-optimization gpyopt tensorboard slurm
Language:Python 136
ohsu-comp-bio / funnel
Funnel is a toolkit for distributed task execution via a simple, standard API.
ga4gh google-cloud gridengine slurm pbs-torque docker aws-batch kubernetes
Language:Go 131
NREL / HPC
A collection of various resources, examples, and executables for the general NREL HPC user community's benefit. Use the following website for accessing documentation.
computing energy high hpc lab laboratory national nrel performance renewable slurm training
Language:Jupyter Notebook 129
SciDAS / slurm-in-docker
Slurm in Docker - Exploring Slurm using CentOS 7 based Docker images
slurm docker rpm-packages centos7 openmpi cluster centos hpc lmod
Language:Shell 129
sylabs / wlm-operator
Singularity implementation of k8s operator for interacting with SLURM.
kubernetes k8s slurm kubernetes-operator singularity hpc
Language:Go 117
neilmunday / slurm-mail
Slurm-Mail is a drop in replacement for Slurm's e-mails to give users much more information about their jobs compared to the standard Slurm e-mails.
email email-template python slurm
Language:Python 112
CLAIRE-Labo / python-ml-research-template
A template for starting reproducible Python machine-learning projects with hardware acceleration. Find an example at https://github.com/CLAIRE-Labo/no-representation-no-trust
docker machine-learning machine-learning-template python python-package-template python-template reproducibility hpc nvidia slurm
Language:Shell 110
mil-ad / stui
A Slurm dashboard for the terminal.
slurm slurm-cluster slurm-utility urwid tui terminal-based python
Language:Python 99
wjwei-handsome / Slurmer
A TUI application for monitoring and managing SLURM jobs.
ratatui slurm slurm-cluster slurm-workload-manager tui
Language:Rust 89
giovtorres / docker-centos7-slurm
Slurm Docker Container on CentOS 7
docker slurm centos7 centos hpc
Language:Python 88