data-parallelism

There are 9 repositories under data-parallelism topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
deep-learning hpc large-scale data-parallelism pipeline-parallelism model-parallelism ai big-model distributed-computing inference heterogeneous-training foundation-models
Language:Python 38517
microsoft / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
billion-parameters compression data-parallelism deep-learning gpu inference machine-learning mixture-of-experts model-parallelism pipeline-parallelism pytorch trillion-parameters zero
Language:Python 34455
cerndb / dist-keras
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
machine-learning deep-learning apache-spark data-parallelism distributed-optimizers keras optimization-algorithms tensorflow data-science hadoop
Language:Python 623
mratsim / weave
A state-of-the-art multithreading runtime: message-passing based, fast, scalable, ultra-low overhead
multithreading runtime message-passing openmp parallelism task-scheduler work-stealing threadpool scheduler fork-join task-parallelism data-parallelism
Language:Nim 535
PaddlePaddle / PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
fleet-api paddlepaddle benchmark distributed-algorithm large-scale model-parallelism data-parallelism pipeline-parallelism cloud paddlecloud elastic lightning pretraining self-supervised-learning unsupervised-learning
Language:Python 436
Oneflow-Inc / libai
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
oneflow nlp deep-learning large-scale data-parallelism model-parallelism distributed-training pipeline-parallelism transformer self-supervised-learning vision-transformer
Language:Python 384
alibaba / EasyParallelLibrary
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
deep-learning data-parallelism model-parallelism pipeline-parallelism memory-efficient distributed-training gpu
Language:Python 259
dkeras
dkeras-project / dkeras
Distributed Keras Engine, Make Keras faster with only one line of code.
machine-learning keras keras-tensorflow distributed-deep-learning deep-learning tensorflow ray python distributed parallel-computing neural-network deep-neural-networks distributed-systems keras-models keras-neural-networks keras-classification-models tensorflow-models distributed-keras-engine plaidml data-parallelism
Language:Python 192
wenwei202 / terngrad
Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)
distributed-training sgd quantization deep-learning deep-neural-networks data-parallelism
Language:Python 180
orkhon
vertexclique / orkhon
Orkhon: ML Inference Framework and Server Runtime
async data-parallelism inference-server machine-learning multiprocessing python3 tensorflow
Language:Rust 146
xrsrke / pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
megatron megatron-lm transformers 3d-parallelism data-parallelism pipeline-parallelism tensor-parallelism model-parallelism zero-1 large-scale-language-modeling huggingface-transformers distributed-optimizers mixture-of-experts moe sequence-parallelism
Language:Python 76
kuixu / keras_multi_gpu
Multi-GPU training for Keras
multi-gpu keras data-parallelism
Language:Python 44
hkproj / pytorch-transformer-distributed
Distributed training (multi-node) of a Transformer model
collective-communication data-parallelism deep-learning distributed-data-parallel distributed-training gradient-accumulation machine-learning model-parallelism pytorch tutorial
Language:Python 32
NERSC / sc23-dl-tutorial
SC23 Deep Learning at Scale Tutorial Material
ai-for-science data-parallelism deep-learning model-parallelism vision-transformers
Language:Python 32
ryantd / veloce
WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.
ray distributed pytorch distributed-computing data-parallelism parameter-server deep-learning model-parallelism sparsity heterogeneity
Language:Python 18
daekeun-ml / sm-distributed-training-step-by-step
This repository provides hands-on labs on PyTorch-based Distributed Training and SageMaker Distributed Training. It is written to make it easy for beginners to get started, and guides you through step-by-step modifications to the code based on the most basic BERT use cases.
data-parallelism distributed-training sagemaker pytorch-ddp
Language:Jupyter Notebook 13
yangyang14641 / Parallel-Matrix-Multiplication-FOX-Algorithm
:coffee:Implement of Parallel Matrix Multiplication Methods Using FOX Algorithm on Peking University's High-performance Computing System
matrix-multiplication parallel-algorithm fox-algorithm data-parallelism algorithm-analysis intel mpi openmp high-performance-computing supercomputing
Language:C 11
namhoonlee / effect-dps-public
Understanding the effects of data parallelism and sparsity on neural network training
data-parallelism sparsity neural-network-training sparse-neural-network
Language:Python 8
AlibabaPAI / FlashModels
Fast and easy distributed model training examples.
distributed-training xla data-parallelism deep-learning fsdp model-parallelism pytorch sequence-parallelism tensor-parallelism zero llm
Language:Python 6
dscpesu / NetTorrent
A decentralized and distributed framework for training DNNs
p2p-networks data-parallelism model-parallelism distributed-computing
Language:Python 4
itzmeanjan / merklize-blake3
OpenCL powered Merklization using BLAKE3
opencl cryptographic-hash-functions blake3 data-parallelism simd merkle-tree binary-merklization
Language:C 4
Oblomov / cldpp
OpenCL Data Parallel Primitives
opencl gpgpu reduction scan data-parallelism
Language:C 4
plerros / helsing
A mostly POSIX-compliant utility that scans a given interval for vampire numbers.
algorithm c posix vampire-number recreational-mathematics recreational math mathematics makefile linked-list checkpoint checkpoint-restart data-parallelism freebsd homebrew linux unrolled-linked-list quicksort vampire number
Language:C 4
zbjob / DiscoPoP
Dependence-Based Code Transformation for Coarse-Grained Parallelism
parallelization data-parallelism code-transformation
Language:C++ 4
explcre / pipeDejavu
pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism
data-parallelism deep-learning differentiable-programming distributed-training dynamic-programming model-parallelism pipeline-parallelism pytorch random-initialization differentiable-dynamic-programming python baysian-optimisation hardware-aware linear-regression parallel-optmization predictive-modeling sampling-methods
Language:Jupyter Notebook 3
LER0ever / HPGO
Development of Project HPGO | Hybrid Parallelism Global Orchestration
pipeline-parallelism data-parallelism distributed-training tensorflow pytorch pipedream gpipe machine-learning model-parallelism rust
3
batch-partitioning
sjlee25 / batch-partitioning
Batch Partitioning for Multi-PE Inference with TVM (2020)
tvm deep-learning data-parallelism dl-compiler dl-optimization inference-optimization
Language:Python 3
AnveshaM / Enhancing-performance-of-big-data-machine-learning-models-on-Google-Cloud-Platform
The project is focused on parallelising pre-processing, measuring and machine learning in the cloud, as well as the evaluation and analysis of the cloud performance.
cache data-parallelism dataproc-clusters google-cloud-platform keras-tensorflow ml model-parallelism rdd google-colaboratory pyspark google-cloud-ai-platform
Language:Jupyter Notebook 2
ashayp22 / monte-carlo-options-simd
SIMD multithreaded Monte Carlo options pricer in Rust 🦀
monte-carlo-simulation options-pricing rust simd data-parallelism rayon
Language:Rust 2
axr6077 / Ray-Trace-Parallelization
Complex ray tracing algorithm optimized by using parallelization over different partitioning schemes and explore the performance gains through grain size and processing units (parameters) over sequential algorithm to render a high resolution image.
data-parallelism parallel-computing cluster-computing mpi optimization-methods partitioning-algorithms rendering-2d-graphics
Language:C++ 2
HiEST / DistMIS
Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation
deep-neural-networks deep-learning tensorflow raylib distributed hyperparameter-tuning data-parallelism medical-image-segmentation 3d-unet ray-tune distributed-hyperparameter-tuning experiment-parallelism
Language:Python 2
MurrellGroup / Conflux.jl
Single-node data parallelism in Julia with CUDA
cuda data-parallelism flux julia nccl
Language:Julia 2
ngrabaskas / Torch-Automatic-Distributed-Neural-Network
Torch Automatic Distributed Neural Network (TorchAD-NN) training library. Built on top of TorchMPI, this module automatically parallelizes neural network training.
data-parallelism model-parallelism torch7 openmpi machine-learning neural-network distributed-machine-learning
Language:Lua 2
oekosheri / pytorch_unet_scaling
Scaling Unet in Pytorch
data-parallelism gpu-parallelization horovod pytorch unet-image-segmentation ddp
Language:Jupyter Notebook 2
axr6077 / Hogdkin-Huxley-Neuron-Model
Sequential and Parallel Implementation of the Hodgkin-Huxley Neuron model.
computational-biology data-parallelism openmpi optimization parallel-computing
Language:C 1
oekosheri / tensorflow_unet_scaling
Scaling Unet in Tensorflow
data-parallelism gpu-parallelism horovod tensorflow unet-image-segmentation mirrored-strategy multi-worker-strategy
Language:Jupyter Notebook 1

data-parallelism

hpcaitech / ColossalAI

microsoft / DeepSpeed

cerndb / dist-keras

mratsim / weave

PaddlePaddle / PaddleFleetX

Oneflow-Inc / libai

alibaba / EasyParallelLibrary

dkeras-project / dkeras

wenwei202 / terngrad

vertexclique / orkhon

xrsrke / pipegoose

kuixu / keras_multi_gpu

hkproj / pytorch-transformer-distributed

NERSC / sc23-dl-tutorial

ryantd / veloce

daekeun-ml / sm-distributed-training-step-by-step

yangyang14641 / Parallel-Matrix-Multiplication-FOX-Algorithm

namhoonlee / effect-dps-public

AlibabaPAI / FlashModels

dscpesu / NetTorrent

itzmeanjan / merklize-blake3

Oblomov / cldpp

plerros / helsing

zbjob / DiscoPoP

explcre / pipeDejavu

LER0ever / HPGO

sjlee25 / batch-partitioning

AnveshaM / Enhancing-performance-of-big-data-machine-learning-models-on-Google-Cloud-Platform

ashayp22 / monte-carlo-options-simd

axr6077 / Ray-Trace-Parallelization

HiEST / DistMIS

MurrellGroup / Conflux.jl

ngrabaskas / Torch-Automatic-Distributed-Neural-Network

oekosheri / pytorch_unet_scaling

axr6077 / Hogdkin-Huxley-Neuron-Model

oekosheri / tensorflow_unet_scaling