data-parallelism

There are 8 repositories under data-parallelism topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
ai big-model data-parallelism deep-learning distributed-computing foundation-models heterogeneous-training hpc inference large-scale model-parallelism pipeline-parallelism
Language:Python 37853
microsoft / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
billion-parameters compression data-parallelism deep-learning gpu inference machine-learning mixture-of-experts model-parallelism pipeline-parallelism pytorch trillion-parameters zero
Language:Python 32636
cerndb / dist-keras
Distributed Deep Learning, with a focus on distributed training, using Keras and Apache Spark.
machine-learning deep-learning apache-spark data-parallelism distributed-optimizers keras optimization-algorithms tensorflow data-science hadoop
Language:Python 624
mratsim / weave
A state-of-the-art multithreading runtime: message-passing based, fast, scalable, ultra-low overhead
data-parallelism fork-join message-passing multithreading openmp parallelism runtime scheduler task-parallelism task-scheduler threadpool work-stealing
Language:Nim 522
PaddlePaddle / PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
fleet-api paddlepaddle benchmark distributed-algorithm large-scale model-parallelism data-parallelism pipeline-parallelism cloud paddlecloud elastic lightning pretraining self-supervised-learning unsupervised-learning
Language:Python 421
Oneflow-Inc / libai
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
oneflow nlp deep-learning large-scale data-parallelism model-parallelism distributed-training pipeline-parallelism transformer self-supervised-learning vision-transformer
Language:Python 376
alibaba / EasyParallelLibrary
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
deep-learning data-parallelism model-parallelism pipeline-parallelism memory-efficient distributed-training gpu
Language:Python 252
dkeras
dkeras-project / dkeras
Distributed Keras Engine, Make Keras faster with only one line of code.
machine-learning keras keras-tensorflow distributed-deep-learning deep-learning tensorflow ray python distributed parallel-computing neural-network deep-neural-networks distributed-systems keras-models keras-neural-networks keras-classification-models tensorflow-models distributed-keras-engine plaidml data-parallelism
Language:Python 192
wenwei202 / terngrad
Ternary Gradients to Reduce Communication in Distributed Deep Learning (TensorFlow)
distributed-training sgd quantization deep-learning deep-neural-networks data-parallelism
Language:Python 181
orkhon
vertexclique / orkhon
Orkhon: ML Inference Framework and Server Runtime
inference-server machine-learning python3 tensorflow async multiprocessing data-parallelism
Language:Rust 145
xrsrke / pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
megatron megatron-lm transformers 3d-parallelism data-parallelism pipeline-parallelism tensor-parallelism model-parallelism zero-1 large-scale-language-modeling huggingface-transformers distributed-optimizers mixture-of-experts moe sequence-parallelism
Language:Python 68
kuixu / keras_multi_gpu
Multi-GPU training for Keras
multi-gpu keras data-parallelism
Language:Python 44
hkproj / pytorch-transformer-distributed
Distributed training (multi-node) of a Transformer model
collective-communication data-parallelism deep-learning distributed-data-parallel distributed-training gradient-accumulation machine-learning model-parallelism pytorch tutorial
Language:Python 18
ryantd / veloce
WIP. Veloce is a low-code Ray-based parallelization library that makes machine learning computation novel, efficient, and heterogeneous.
ray distributed pytorch distributed-computing data-parallelism parameter-server deep-learning model-parallelism sparsity heterogeneity
Language:Python 18
daekeun-ml / sm-distributed-training-step-by-step
This repository provides hands-on labs on PyTorch-based Distributed Training and SageMaker Distributed Training. It is written to make it easy for beginners to get started, and guides you through step-by-step modifications to the code based on the most basic BERT use cases.
data-parallelism distributed-training sagemaker pytorch-ddp
Language:Jupyter Notebook 12
yangyang14641 / Parallel-Matrix-Multiplication-FOX-Algorithm
:coffee:Implement of Parallel Matrix Multiplication Methods Using FOX Algorithm on Peking University's High-performance Computing System
matrix-multiplication parallel-algorithm fox-algorithm data-parallelism algorithm-analysis intel mpi openmp high-performance-computing supercomputing
Language:C 12
namhoonlee / effect-dps-public
Understanding the effects of data parallelism and sparsity on neural network training
data-parallelism sparsity neural-network-training sparse-neural-network
Language:Python 8
dscpesu / NetTorrent
A decentralized and distributed framework for training DNNs
p2p-networks data-parallelism model-parallelism distributed-computing
Language:Python 4
itzmeanjan / merklize-blake3
OpenCL powered Merklization using BLAKE3
opencl cryptographic-hash-functions blake3 data-parallelism simd merkle-tree binary-merklization
Language:C 4
Oblomov / cldpp
OpenCL Data Parallel Primitives
data-parallelism gpgpu opencl reduction scan
Language:C 4
zbjob / DiscoPoP
Dependence-Based Code Transformation for Coarse-Grained Parallelism
parallelization data-parallelism code-transformation
Language:C++ 4
explcre / pipeDejavu
pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism
data-parallelism deep-learning differentiable-programming distributed-training dynamic-programming model-parallelism pipeline-parallelism pytorch random-initialization differentiable-dynamic-programming python baysian-optimisation hardware-aware linear-regression parallel-optmization predictive-modeling sampling-methods
Language:Jupyter Notebook 3
LER0ever / HPGO
Development of Project HPGO | Hybrid Parallelism Global Orchestration
data-parallelism distributed-training gpipe machine-learning model-parallelism pipedream pipeline-parallelism pytorch rust tensorflow
3
plerros / helsing
A mostly POSIX-compliant utility that scans a given interval for vampire numbers.
algorithm c posix vampire-number recreational-mathematics recreational math mathematics makefile linked-list checkpoint checkpoint-restart data-parallelism freebsd homebrew linux unrolled-linked-list quicksort vampire number
Language:C 3
batch-partitioning
sjlee25 / batch-partitioning
Batch Partitioning for Multi-PE Inference with TVM (2020)
tvm deep-learning data-parallelism dl-compiler dl-optimization inference-optimization
Language:Python 3
AnveshaM / Enhancing-performance-of-big-data-machine-learning-models-on-Google-Cloud-Platform
The project is focused on parallelising pre-processing, measuring and machine learning in the cloud, as well as the evaluation and analysis of the cloud performance.
cache data-parallelism dataproc-clusters google-cloud-platform keras-tensorflow ml model-parallelism rdd google-colaboratory pyspark google-cloud-ai-platform
Language:Jupyter Notebook 2
ashayp22 / monte-carlo-options-simd
SIMD multithreaded Monte Carlo options pricer in Rust 🦀
monte-carlo-simulation options-pricing rust simd data-parallelism rayon
Language:Rust 2
axr6077 / Ray-Trace-Parallelization
Complex ray tracing algorithm optimized by using parallelization over different partitioning schemes and explore the performance gains through grain size and processing units (parameters) over sequential algorithm to render a high resolution image.
data-parallelism parallel-computing cluster-computing mpi optimization-methods partitioning-algorithms rendering-2d-graphics
Language:C++ 2
HiEST / DistMIS
Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation
deep-neural-networks deep-learning tensorflow raylib distributed hyperparameter-tuning data-parallelism medical-image-segmentation 3d-unet ray-tune distributed-hyperparameter-tuning experiment-parallelism
Language:Python 2
ngrabaskas / Torch-Automatic-Distributed-Neural-Network
Torch Automatic Distributed Neural Network (TorchAD-NN) training library. Built on top of TorchMPI, this module automatically parallelizes neural network training.
data-parallelism model-parallelism torch7 openmpi machine-learning neural-network distributed-machine-learning
Language:Lua 2
oekosheri / pytorch_unet_scaling
Scaling Unet in Pytorch
data-parallelism gpu-parallelization horovod pytorch unet-image-segmentation ddp
Language:Jupyter Notebook 2
axr6077 / Hogdkin-Huxley-Neuron-Model
Sequential and Parallel Implementation of the Hodgkin-Huxley Neuron model.
computational-biology data-parallelism openmpi optimization parallel-computing
Language:C 1
diptorupd / numba-dpex
A SYCL-like kernel compiler for Python
compiler data-parallelism jit numba python sycl
Language:Python 1
oekosheri / tensorflow_unet_scaling
Scaling Unet in Tensorflow
data-parallelism gpu-parallelism horovod tensorflow unet-image-segmentation mirrored-strategy multi-worker-strategy
Language:Jupyter Notebook 1
oriolaranda / DistMIS
Official Repository for the paper: Distributing Deep Learning Hyperparameter Tuning for 3D Medical Image Segmentation
deep-learning deep-neural-networks 3d-unet medical-image-segmentation hyperparameter-tuning distributed data-parallelism raylib ray-tune tensorflow distributed-hyperparameter-tuning experiment-parallelism
Language:Python 1
Sujith013 / Binary-Classification-using-Machine-Learning-and-Data-parallelism
Binary data classification using TensorFlow and Keras in python and achieving data parallelism using MPI
binary-classification data-augmentation data-parallelism deep-learning jupyter-notebook keras machine-learning mpi mpi4py parallel-computing pet-classifier python tensorflow
Language:Jupyter Notebook 1

data-parallelism

hpcaitech / ColossalAI

microsoft / DeepSpeed

cerndb / dist-keras

mratsim / weave

PaddlePaddle / PaddleFleetX

Oneflow-Inc / libai

alibaba / EasyParallelLibrary

dkeras-project / dkeras

wenwei202 / terngrad

vertexclique / orkhon

xrsrke / pipegoose

kuixu / keras_multi_gpu

hkproj / pytorch-transformer-distributed

ryantd / veloce

daekeun-ml / sm-distributed-training-step-by-step

yangyang14641 / Parallel-Matrix-Multiplication-FOX-Algorithm

namhoonlee / effect-dps-public

dscpesu / NetTorrent

itzmeanjan / merklize-blake3

Oblomov / cldpp

zbjob / DiscoPoP

explcre / pipeDejavu

LER0ever / HPGO

plerros / helsing

sjlee25 / batch-partitioning

AnveshaM / Enhancing-performance-of-big-data-machine-learning-models-on-Google-Cloud-Platform

ashayp22 / monte-carlo-options-simd

axr6077 / Ray-Trace-Parallelization

HiEST / DistMIS

ngrabaskas / Torch-Automatic-Distributed-Neural-Network

oekosheri / pytorch_unet_scaling

axr6077 / Hogdkin-Huxley-Neuron-Model

diptorupd / numba-dpex

oekosheri / tensorflow_unet_scaling

oriolaranda / DistMIS

Sujith013 / Binary-Classification-using-Machine-Learning-and-Data-parallelism