huangwei021230

InternLM

Official release of InternLM2.5 base and chat models. 1M context support

arena-hard-auto

Arena-Hard-Auto: An automatic LLM benchmark.

TransformerCompression

For releasing code related to compression methods for transformers, accompanying our publications

FlagAttention

A collection of memory efficient attention operators implemented in the Triton language.

storm

An LLM-powered knowledge curation system that researches a topic and generates a full-length report with citations.

RAG-Retrieval

Unify Efficient Fine-tuning of RAG Retrieval, including Embedding, ColBERT,Cross Encoder

CUDA_Freshman

FasterTransformer

Transformer related optimization, including BERT, GPT

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

calm

CUDA/Metal accelerated language model inference

dgl

Python package built to ease deep learning on graph, on top of existing DL frameworks.

Liger-Kernel

Efficient Triton Kernels for LLM Training

EQ-Bench

A benchmark for emotional intelligence in large language models

OmniQuant

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

AutoAWQ_kernels

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

flashinfer

FlashInfer: Kernel Library for LLM Serving

QuaRot

Code for QuaRot, an end-to-end 4-bit inference of large language models.

any-precision-llm

[ICML 2024 Oral] Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs

T-MAC

Low-bit LLM inference on CPU with lookup table

TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library

ANT-Quantization

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

GPTQ-triton

GPTQ inference Triton kernel

Qwen2

Qwen2 is the large language model series developed by Qwen team, Alibaba Cloud.

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

llm-awq

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration