Shigangli

Magicube

Magicube is a high-performance library for quantized sparse matrix operations (SpMM and SDDMM) of deep learning on Tensor Cores.

Chimera

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines.

Ok-Topk

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

eager-SGD

Eager-SGD is a decentralized asynchronous SGD. It utilizes novel partial collectives operations to accumulate the gradients across all the processes.

COMPI

Cache-oblivious MPI all-to-all communications based on Morton order

DNN-cpp-proxies

C++/MPI proxies for distributed training of deep neural networks.

akg

AKG (Auto Kernel Generator) is an optimizer for operators in Deep Learning Networks, which provides the ability to automatically fuse ops with specific patterns.

bigbird

Transformers for Longer Sequences

bigbird-1

Google's BigBird (Jax/Flax & PyTorch) @ 🤗Transformers

brian2

Brian is a free, open source simulator for spiking neural networks.

ColossalAI

Making large AI models cheaper, faster and more accessible

CuAssembler

An unofficial cuda assembler, for all generations of SASS, hopefully ：）

dace

DaCe - Data Centric Parallel Programming

dlrm

An implementation of a deep learning recommendation model (DLRM)

examples

A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

flax

Flax is a neural network library for JAX that is designed for flexibility.

legion

The Legion Parallel Programming System

longformer

Longformer: The Long-Document Transformer

mindspore

MindSpore is a new open source deep learning training/inference framework that could be used for mobile, edge and cloud scenarios.

minGPT

A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

Nystromformer

oneDNN

oneAPI Deep Neural Network Library (oneDNN)

p4app-switchML

Switch ML Application

ROCm

ROCm - Open Source Platform for HPC and Ultrascale GPU Computing

shigangli.github.io

Homepage of Shigang Li https://shigangli.github.io/

sparsegpt

Code for the paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".

TensorRT_Tutorial

vision_transformer

XiangShan

Open-source high-performance RISC-V processor

yapf

A formatter for Python files