pipeline-parallelism

There are 5 repositories under pipeline-parallelism topic.

ColossalAI
hpcaitech / ColossalAI
Making large AI models cheaper, faster and more accessible
ai big-model data-parallelism deep-learning distributed-computing foundation-models heterogeneous-training hpc inference large-scale model-parallelism pipeline-parallelism
Language:Python 41228
deepspeedai / DeepSpeed
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
billion-parameters compression data-parallelism deep-learning gpu inference machine-learning mixture-of-experts model-parallelism pipeline-parallelism pytorch trillion-parameters zero
Language:Python 40633
bigscience-workshop / petals
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
bloom chatbot deep-learning distributed-systems falcon gpt guanaco language-models large-language-models llama machine-learning mixtral neural-networks nlp pipeline-parallelism pretrained-models pytorch tensor-parallelism transformer volunteer-computing
Language:Python 9830
torchgpipe
kakaobrain / torchgpipe
A GPipe implementation in PyTorch
checkpointing deep-learning gpipe model-parallelism parallelism pipeline-parallelism pytorch
Language:Python 857
Coobiw / MPP-LLaVA
Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
deepspeed fine-tuning mllm model-parallel multimodal-large-language-models pipeline-parallelism pretraining qwen video-language-model video-large-language-models
Language:Jupyter Notebook 478
PaddlePaddle / PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
fleet-api paddlepaddle benchmark distributed-algorithm large-scale model-parallelism data-parallelism pipeline-parallelism cloud paddlecloud elastic lightning pretraining self-supervised-learning unsupervised-learning
Language:Python 474
InternLM / InternEvo
InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.
deepspeed-ulysses gemma internlm internlm2 llama3 llava llm-framework llm-training multi-modal pipeline-parallelism ring-attention sequence-parallelism tensor-parallelism transformers-models zero3 910b flash-attention pytorch
Language:Python 410
Oneflow-Inc / libai
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
oneflow nlp deep-learning large-scale data-parallelism model-parallelism distributed-training pipeline-parallelism transformer self-supervised-learning vision-transformer
Language:Python 407
alibaba / EasyParallelLibrary
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
deep-learning data-parallelism model-parallelism pipeline-parallelism memory-efficient distributed-training gpu
Language:Python 270
Shenggan / awesome-distributed-ml
A curated list of awesome projects and papers for distributed training or inference
deep-learning distributed-systems high-performance-computing machine-learning model-parallelism pipeline-parallelism
249
torchpipe / torchpipe
Serving Inside Pytorch
deployment inference pipeline-parallelism serving tensorrt triton-inference-server ray pytorch torch2trt serve llm-serving
Language:C++ 164
ai-decentralized / BloomBee
Decentralized LLMs fine-tuning and inference with offloading
deep-learning distributed-systems llama machine-learning pipeline-parallelism pytorch tensor-parallelism
Language:Python 102
xrsrke / pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
megatron megatron-lm transformers 3d-parallelism data-parallelism pipeline-parallelism tensor-parallelism model-parallelism zero-1 large-scale-language-modeling huggingface-transformers distributed-optimizers mixture-of-experts moe sequence-parallelism
Language:Python 87
AlibabaPAI / DAPPLE
An Efficient Pipelined Data Parallel Approach for Training Large Model
pipeline-parallelism hybrid-parallelism distribution-strategy-planner
Language:Python 76
ParCIS / Chimera
Chimera: bidirectional pipeline parallelism for efficiently training large-scale models.
transformers distributed-deep-learning pipeline-parallelism
Language:Python 67
MLSysU / TD-Pipe
A Throughput-Optimized Pipeline Parallel Inference System for Large Language Models
high-throughput llm-inference pipeline-parallelism vllm
Language:Python 44
saareliad / FTPipe
FTPipe and related pipeline model parallelism research.
deep-neural-networks distributed-training fine-tuning nlp pipeline-parallelism t5
Language:Python 43
gty111 / gLLM
gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
llm-inference pipeline-parallelism llm-serving token-throttling qwen3 tensor-parallelism continuous-batching pagedattention chunked-prefill
Language:Python 42
nawnoes / pytorch-gpt-x
Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.
gpt pytorch transformer pipeline-parallelism deepspeed
Language:Python 32
fanpu / DynPartition
Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks
dynamic-neural-network machine-learning model-parallelism neural-networks pipeline-parallelism pytorch reinforcement-learning scheduling treelstm dynpartition
Language:Python 7
garg-aayush / model-parallelism
Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)
gpipe model-parallelism pipeline-parallelism pytorch
Language:Python 5
torchpipe.github.io
torchpipe / torchpipe.github.io
Docs for torchpipe: https://github.com/torchpipe/torchpipe
deployment inference pipeline-parallelism pytorch serving tensorrt
Language:MDX 4
explcre / pipeDejavu
pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism
data-parallelism deep-learning differentiable-programming distributed-training dynamic-programming model-parallelism pipeline-parallelism pytorch random-initialization differentiable-dynamic-programming python baysian-optimisation hardware-aware linear-regression parallel-optmization predictive-modeling sampling-methods
Language:Jupyter Notebook 3
LER0ever / HPGO
Development of Project HPGO | Hybrid Parallelism Global Orchestration
pipeline-parallelism data-parallelism distributed-training tensorflow pytorch pipedream gpipe machine-learning model-parallelism rust
3
joe0731 / hf_vram_calc
A CLI tool for estimating GPU VRAM requirements for Hugging Face models, supporting various data types, parallelization strategies, and fine-tuning scenarios like LoRA.
hugging-face-transformers huggingface huggingface-datasets huggingface-models memory-estimation vram gpu-memory pipeline-parallelism vram-monitoring vram-calculator vram-memory-estimation
Language:Python 1
1set-t / ai-model
Industrial-grade weather visualization system that transforms AI model predictions into professional meteorological plots, emphasizing operational forecasting capabilities.
data-parallelism data-science deep-learning face-detection image-classification large-language-models large-scale llms mlops multimodal object-detection pipeline-parallelism pose-estimation python
liam8421 / faster-llm
🚀 Accelerate LLM training with Fast-LLM, an open-source library for high-speed, scalable, and flexible model development.
bloom distributed-systems gateway gemma large-language-models llama llama3 llm llmops mcp-gateway mistral neural-networks openai pipeline-parallelism text-to-speech unsloth voice-cloning volunteer-computing
Language:Python
sparklerz / multigpu-llm-finetuning
This repository showcases hands-on projects leveraging distributed multi-GPU training to fine-tune large language models (LLMs).
ddp deepspeed distributed-training fsdp llm mlflow mlops mosaicml pipeline-parallelism pytorch ray ray-train ray-tune wandb
Language:Python