There are 3 repositories under pipeline-parallelism topic.
Making large AI models cheaper, faster and more accessible
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
A GPipe implementation in PyTorch
飞桨大模型开发套件,提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
LiBai(李白): A Toolbox for Large-Scale Distributed Parallel Training
Easy Parallel Library (EPL) is a general and efficient deep learning framework for distributed model training.
A curated list of awesome projects and papers for distributed training or inference
An Efficient Pipelined Data Parallel Approach for Training Large Model
Implementation of autoregressive language model using improved Transformer and DeepSpeed pipeline parallelism.
Official implementation of DynPartition: Automatic Optimal Pipeline Parallelism of Dynamic Neural Networks over Heterogeneous GPU Systems for Inference Tasks
Model parallelism for NN architectures with skip connections (eg. ResNets, UNets)
Docs for torchpipe: https://github.com/torchpipe/torchpipe
pipeDejavu: Hardware-aware Latency Predictable, Differentiable Search for Faster Config and Convergence of Distributed ML Pipeline Parallelism