brisker

llama.cpp

LLM inference in C/C++

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

mlc-llm

Universal LLM Deployment Engine with ML Compilation

lantern

Lantern官方版本下载 蓝灯 翻墙 代理 科学上网 外网 加速器 梯子 路由 - Быстрый, надежный и безопасный доступ к открытому интернету - lantern proxy vpn censorship-circumvention censorship gfw accelerator پراکسی لنترن، ضدسانسور، امن، قابل اعتماد و پرسرعت

flash-attention

Fast and memory-efficient exact attention

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

lm-evaluation-harness

A framework for few-shot evaluation of language models.

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Baichuan2

A series of large language models developed by Baichuan Intelligent Technology

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

GPTQ-for-LLaMa

4 bits quantization of LLaMA using GPTQ

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

test

Measuring Massive Multitask Language Understanding | ICLR 2021

ToMe

A method to increase the speed and lower the memory footprint of existing vision transformers.

Awesome-LLM-Compression

Awesome LLM compression research papers and tools.

llama-chat

Chat with Meta's LLaMA models at home made easy

MQBench

Model Quantization Benchmark

QuIP

Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"

hessian

hessian in pytorch

QUIK

Repository for the QUIK project, enabling the use of 4bit kernels for generative inference

torch-int

This repository contains integer operators on GPUs for PyTorch.

INT8-Flash-Attention-FMHA-Quantization

PB-LLM

PB-LLM: Partially Binarized Large Language Models

ANT-Quantization

Outlier_Suppression_Plus

Official implementation of the EMNLP23 paper: Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling

ReSTE

Official implementation of Rectified Straight Through Estimator (ReSTE).

AFPQ

AFPQ code implementation

llm-mixed-q

mixed-precision quantization for LLMs