imisszxq

mscclpp

MSCCL++: A GPU-driven communication stack for scalable AI applications

ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

QuaRot

Code for QuaRot, an end-to-end 4-bit inference of large language models.

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

llama.cpp

LLM inference in C/C++

sarathi-serve

A low-latency & high-throughput serving engine for LLMs

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

cuda-training-series

Training materials associated with NVIDIA's CUDA Training Series (www.olcf.ornl.gov/cuda-training-series/)

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

DistServe

Disaggregated serving system for Large Language Models (LLMs).

lectures

Material for cuda-mode lectures

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

xcpc-algorithm-templates

XCPC/ICPC/CCPC 算法模板

sglang

SGLang is yet another fast serving framework for large language models and vision language models.

CUDATutorial

A CUDA tutorial to make people learn CUDA program from 0

maxas

Assembler for NVIDIA Maxwell architecture

How_to_optimize_in_GPU

This is a series of GPU optimization topics. Here we will introduce how to optimize the CUDA kernel in detail. I will introduce several basic kernel optimizations, including: elementwise, reduce, sgemv, sgemm, etc. The performance of these kernels is basically at or near the theoretical limit.

Skywork-MoE

Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Models

flash-llm

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity

fp6_llm

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

flashinfer

FlashInfer: Kernel Library for LLM Serving

qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

cutlass-kernels

llm.c

LLM training in simple, raw C/CUDA

MatmulTutorial

A Easy-to-understand TensorOp Matmul Tutorial

glake

GLake: optimizing GPU memory management and IO transmission.

CUDATutorial

A self-learning tutorail for CUDA High Performance Programing.