yingfeng

PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

Anima

33B Chinese LLM, DPO QLORA, 100K context, AirLLM 70B inference with single 4GB GPU

blazingmq

A modern high-performance open source message queuing system

infinity

The AI-native database built for LLM applications, providing incredibly fast hybrid search of dense embedding, sparse embedding, tensor, and full-text

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

LLMCompiler

[ICML 2024] LLMCompiler: An LLM Compiler for Parallel Function Calling

flashinfer

FlashInfer: Kernel Library for LLM Serving

SqueezeLLM

[ICML 2024] SqueezeLLM: Dense-and-Sparse Quantization

Qwen-TensorRT-LLM

FalkorDB

A super fast Graph Database uses GraphBLAS under the hood for its sparse adjacency matrix graph representation. Our goal is to provide the best Knowledge Graph for LLM (GraphRAG).

SwiftInfer

Efficient AI Inference & Serving

H2O

[NeurIPS'23] H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models.

neural-speed

An innovative library for efficient LLM inference via low-bit quantization

xFasterTransformer

PainlessInferenceAcceleration

KVQuant

KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization

concurrent_deferred_rc

Concurrent Deferred Reference Counting

relbench

RelBench: Relational Deep Learning Benchmark

ServerlessLLM

Fast, easy and cost-efficient multi-LLM serving.

SpotServe

SpotServe: Serving Generative Large Language Models on Preemptible Instances

MSVBASE

MSVBASE is a system that efficiently supports complex queries of both approximate similarity search and relational operators. It integrates high-dimensional vector indices into PostgreSQL, a relational database to facilitate complex approximate similarity queries.

BEBR

Official code for "Binary embedding based retrieval at Tencent"

tunnel

Tunnel is a Pipeline Execution Engine based on C++20 coroutine

fast-multi-join-sketch

Fast Cardinality Estimation of Multi-Join Queries Using Sketches

constrainedANN

bqf

Implementation of a Backpack Quotient Filter

exaloglog-paper

ExaLogLog: Space-Efficient and Practical Approximate Distinct Counting up to the Exa-Scale

tabby-dbos

verlib

Hetu

A high-performance distributed deep learning system targeting large-scale and automated distributed training.