sheng-qin

ktransformers

A Flexible Framework for Experiencing Cutting-edge LLM Inference Optimizations

flux

Official inference repo for FLUX.1 models

Qwen-TensorRT-LLM

evalscope

A streamlined and customizable framework for efficient large model evaluation and performance benchmarking

LLM-Viewer

Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.

sglang

SGLang is yet another fast serving framework for large language models and vision language models.

LLMRoofline

Compare different hardware platforms via the Roofline Model for LLM inference tasks.

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

AutoGPTQ

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

GPTQ-for-LLaMa

4 bits quantization of LLaMA using GPTQ

AutoFP8

bitsandbytes

Accessible large language models via k-bit quantization for PyTorch.

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

llm-numbers

Numbers every LLM developer should know

composable_kernel

Composable Kernel: Performance Portable Programming Model for Machine Learning Tensor Operators

ThunderKittens

Tile primitives for speedy kernels

xDiT

xDiT: A Scalable Inference Engine for Diffusion Transformers (DiTs) on multi-GPU Clusters

Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

minisora

MiniSora: A community aims to explore the implementation path and future development direction of Sora.

Latte

Latte: Latent Diffusion Transformer for Video Generation.

Open-Sora-Plan

This project aim to reproduce Sora (Open AI T2V model), we wish the open source community contribute to this project.

PixArt-sigma

PixArt-Σ: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation

QPyTorch

Low Precision Arithmetic Simulation in PyTorch

calculon

brevitas

Brevitas: neural network quantization in PyTorch

fp6_llm

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

cccl

CUDA Core Compute Libraries

Zelda64Recomp

Static recompilation of Majora's Mask (and soon Ocarina of Time) for PC (Windows/Linux)

N64Recomp

Tool to statically recompile N64 games into native executables