ymjiang

S-LoRA

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

MS-AMP

Microsoft Automatic Mixed Precision Library

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

flash-attention

Fast and memory-efficient exact attention

trlx

A repo for distributed training of language models with Reinforcement Learning via Human Feedback (RLHF)

Prompt-Engineering-Guide

🐙 Guides, papers, lecture, notebooks and resources for prompt engineering

nanoGPT

The simplest, fastest repository for training/finetuning medium-sized GPTs.

NM-sparsity

PiPPy

Pipeline Parallelism for PyTorch

veGiantModel

pcm

Intel® Performance Counter Monitor (Intel® PCM)

fedlearner

A multi-party collaborative machine learning framework

Megatron-LM

Ongoing research training transformer models at scale

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

attention-is-all-you-need-pytorch

A PyTorch implementation of the Transformer model in "Attention is All You Need".

incubator-mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

byteps

A high performance and generic framework for distributed DNN training

ps-lite

A lightweight parameter server interface

horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.