Xu-Chen

flute

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

flashinfer

FlashInfer: Kernel Library for LLM Serving

Quest

[ICML 2024] Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference

TLLM_QMM

TLLM_QMM strips the implementation of quantized kernels of Nvidia's TensorRT-LLM, removing NVInfer dependency and exposes ease of use Pytorch module. We modified the dequantation and weight preprocessing to align with popular quantization alogirthms such as AWQ and GPTQ, and combine them with new FP8 quantization.

MobileLLM

MobileLLM Optimizing Sub-billion Parameter Language Models for On-Device Use Cases. In ICML 2024.

llm-compressor

GPTModels.nvim

GPTModels - a multi model, window based LLM AI plugin for neovim, with an emphasis on stability and clean code

GPTQModel

An easy-to-use LLM quantization and inference toolkit based on GPTQ algorithm (weight-only quantization).

QQQ

QQQ is an innovative and hardware-optimized W4A8 quantization solution.

ocular

AI Powered Search and Chat for Orgs - Think ChatGPT meets Google Search but powered by your data.

mistral-inference

Official inference library for Mistral models

farfalle

🔍 AI search engine - self-host with local or cloud LLMs

Sparse-Marlin

EAGLE

Official Implementation of EAGLE-1 and EAGLE-2

aspoem

Learn Chinese Poetry With AsPoem.com

DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Perplexica

Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

Eurus

ollama

Get up and running with Llama 3, Mistral, Gemma 2, and other large language models.

clarity-ai

A simple Perplexity AI clone.

aphrodite-engine

PygmalionAI's large-scale inference engine

exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

nm-vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

AutoGPTQ

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

Qwen2

Qwen2 is the large language model series developed by Qwen team, Alibaba Cloud.

llamafia.github

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

openai-scala-client

Scala client for OpenAI API