mgoin

flash-attention

Fast and memory-efficient exact attention

shadPS4

PS4 emulator for Windows,Linux,MacOS

xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.

cutlass

CUDA Templates for Linear Algebra Subroutines

Liger-Kernel

Efficient Triton Kernels for LLM Training

ThunderKittens

Tile primitives for speedy kernels

ao

PyTorch native quantization and sparsity for training and inference

evalplus

Rigourous evaluation of LLM-synthesized code - NeurIPS 2023 & COLM 2024

dune

A shell🐚 by the beach🏖️!

llm-compressor

Transformers-compatible library for applying various compression algorithms to LLMs for optimized deployment with vLLM

Nanoflow

A throughput-oriented high-performance serving framework for LLMs

mirage

Mirage: Automatically Generating Fast GPU Kernels without Programming in Triton/CUDA

TensorRT-Model-Optimizer

TensorRT Model Optimizer is a unified library of state-of-the-art model optimization techniques such as quantization, pruning, distillation, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed on NVIDIA GPUs.

depyf

depyf is a tool to help you understand and adapt to PyTorch compiler torch.compile.

composable_kernel

Composable Kernel: Performance Portable Programming Model for Machine Learning Tensor Operators

Minitron

A family of compressed models obtained via pruning and knowledge distillation

flute

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

gemlite

Simple and fast low-bit matmul kernels in CUDA / Triton

GPTQModel

Production ready LLM model compression/quantization toolkit with accelerated inference support for both cpu/gpu via HF, vLLM, and SGLang.

TEAL

cold-compress

Cold Compress is a hackable, lightweight, and open-source toolkit for creating and benchmarking cache compression methods built on top of GPT-Fast, a simple, PyTorch-native generation codebase.

FLASHNN

TensorRT-Incubator

Experimental projects related to TensorRT

Sparse-Marlin

Boosting 4-bit inference kernels with 2:4 Sparsity

compressed-tensors

A safetensors extension to efficiently store sparse quantized tensors on disk

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

mistral-evals

torch_cgx

Pytorch distributed backend extension with compression support

SPP

[ICML 2024] SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models

py-codegen