wzh99

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

mlc-llm

Universal LLM Deployment Engine with ML Compilation

flash-attention

Fast and memory-efficient exact attention

mamba

Mamba SSM architecture

LLMSurvey

The official GitHub page for the survey paper "A Survey of Large Language Models".

FlexiGen

Running large language models on a single GPU for throughput-oriented scenarios.

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

sglang

SGLang is a fast serving framework for large language models and vision language models.

llm-numbers

Numbers every LLM developer should know

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

S-LoRA

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

how-to-optim-algorithm-in-cuda

how to optimize some algorithm in cuda.

flashinfer

FlashInfer: Kernel Library for LLM Serving

punica

Serving multiple LoRA finetuned LLM as one

How_to_optimize_in_GPU

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

LLMSys-PaperList

Large Language Model (LLM) Systems Paper List

Qwen-TensorRT-LLM

rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

ByteTransformer

optimized BERT transformer inference on NVIDIA GPU. https://arxiv.org/abs/2210.03052

compiler-and-arch

A list of tutorials, paper, talks, and open-source projects for emerging compiler and architecture

glake

GLake: optimizing GPU memory management and IO transmission.

flux

A fast communication-overlapping library for tensor parallelism on GPUs.

flash-llm

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity

triton-shared

Shared Middle-Layer for Triton Compilation

InfiniTensor

M3DM

TileFlow

TileFlow is a performance analysis tool based on Timeloop for fusion dataflows

MAGIS

MAGIS: Memory Optimization via Coordinated Graph Transformation and Scheduling for DNN (ASPLOS'24)

APR

[IJCAI 23] APR: Online Distant Point Cloud Registration Through Aggregated Point Cloud Reconstruction