Shigang Li's repositories
Ok-Topk
Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.
DNN-cpp-proxies
C++/MPI proxies for distributed training of deep neural networks.
bigbird-1
Google's BigBird (Jax/Flax & PyTorch) @ 🤗Transformers
bolt
10x faster matrix and vector operations.
ColossalAI
Making large AI models cheaper, faster and more accessible
CuAssembler
An unofficial cuda assembler, for all generations of SASS, hopefully :)
longformer
Longformer: The Long-Document Transformer
p4app-switchML
Switch ML Application
shigangli.github.io
Homepage of Shigang Li https://shigangli.github.io/
sparsegpt
Code for the paper "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot".