Tom-CaoZH

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

ThunderKittens

Tile primitives for speedy kernels

nvbandwidth

A tool for bandwidth measurements on NVIDIA GPUs.

qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

MAGIS

MAGIS: Memory Optimization via Coordinated Graph Transformation and Scheduling for DNN (ASPLOS'24)

DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

touying

Touying is a powerful package for creating presentation slides in Typst.

Honeycomb

Component-Model Framework in C++

foyer

Hybrid memory and disk cache in Rust

nimble

New file format for storage of large columnar datasets.

SnapKV

Sequoia

scalable and robust tree-based speculative decoding algorithm

TriForce

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding

LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

SAS-Cache

[MSST '24] SAS-Cache: A Semantic-Aware Secondary Cache for LSM-based Key-Value Stores

prophet-rocksdb

[MSST '24] Prophet: Optimizing LSM-Based Key-Value Store on ZNS SSDs with File Lifetime Prediction and Compaction Compensation.

DeepCache

[CVPR 2024] DeepCache: Accelerating Diffusion Models for Free

llama3

The official Meta Llama 3 GitHub site

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

calm

CUDA/Metal accelerated language model inference

llm.c

LLM training in simple, raw C/CUDA

CacheGen

fastmoe

A fast MoE impl for PyTorch

lut-gemm

llamafile

Distribute and run LLMs with a single file.

MiniCPM

MiniCPM-2B: An end-side LLM outperforms Llama2-13B.

libbf

:dart: Bloom filters for C++11

FrozenHot

ragflow

RAGFlow is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding.

Bamboo

Bamboo-7B Large Language Model