Uranus's starred repositories
Awesome-LLM-Inference
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
MInference
To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
ringattention
Transformers with Arbitrarily Large Context
ring-flash-attention
Ring attention implementation with flash attention
ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
AI-Software-Startups
A Survey of AI startups
Consistency_LLM
[ICML 2024] CLLMs: Consistency Large Language Models
long-context-attention
Sequence Parallel Attention for Long Context LLM Model Training and Inference
MatmulTutorial
A Easy-to-understand TensorOp Matmul Tutorial
scattermoe
Triton-based implementation of Sparse Mixture of Experts.
sarathi-serve
A low-latency & high-throughput serving engine for LLMs
libflash_attn
Standalone Flash Attention v2 kernel without libtorch dependency
SwiftTransformer
High performance Transformer implementation in C++.
Odysseus-Transformer
Odysseus: Playground of LLM Sequence Parallelism