Zehaos

ColossalAI

Making large AI models cheaper, faster and more accessible

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

mojo

The Mojo Programming Language

flash-attention

Fast and memory-efficient exact attention

Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.

qlora

QLoRA: Efficient Finetuning of Quantized LLMs

DeepSpeedExamples

Example models using DeepSpeed

UniAD

[CVPR 2023 Best Paper] Planning-oriented Autonomous Driving

alpa

Training and serving large-scale neural networks with auto parallelization.

optimum

🚀 Accelerate training and inference of 🤗 Transformers and 🤗 Diffusers with easy to use hardware optimization tools

lion-pytorch

🦁 Lion, new optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam(w), in Pytorch

kernl

Kernl lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.

SparK

[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"

how-to-optim-algorithm-in-cuda

how to optimize some algorithm in cuda.

python3-source-code-analysis

《Python 3 源码剖析》

eRPC

Efficient RPCs for datacenter networks

PatrickStar

PatrickStar enables Larger, Faster, Greener Pretrained Models for NLP and democratizes AI for everyone.

BMTrain

Efficient Training (including pre-training and fine-tuning) for Big Models

attorch

A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.

dino-vit-features

Official implementation for the paper "Deep ViT Features as Dense Visual Descriptors".

algorithmic-efficiency

MLCommons Algorithmic Efficiency is a benchmark and competition measuring neural network training speedups due to algorithmic improvements in both training algorithms and models.

Pytorch-PCGrad

Pytorch reimplementation for "Gradient Surgery for Multi-Task Learning"

grain

infinity

A lightweight C++ RDMA library for InfiniBand networks.

sagemaker-debugger

Amazon SageMaker Debugger provides functionality to save tensors during training of machine learning jobs and analyze those tensors

mtm

MTM Masked Trajectory Models for Prediction, Representation, and Control.

slapo

A schedule language for large model training

SHARK-Turbine

Unified compiler/runtime for interfacing with PyTorch Dynamo.

rotograd

Official Pytorch's implementation of RotoGrad

rdmapp

C++ interfaces for RDMA access