iquitap

ServerlessLLM

Serverless LLM Serving for Everyone

Cute-Learning

Examples of CUDA implementations by Cutlass CuTe

torchtitan

A native PyTorch Library for large model training

TidalDecode

TidalDecode: A Fast and Accurate LLM Decoding with Position Persistent Sparse Attention

ParrotServe

[OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable

splitwise-sim

LLM serving cluster simulator

Awesome-LLM-Strawberry

A collection of LLM papers, blogs, and projects, with a focus on OpenAI o1 and reasoning techniques.

tlm

AIOS

AIOS: LLM Agent Operating System

Nanoflow

A throughput-oriented high-performance serving framework for LLMs

triton-shared

Shared Middle-Layer for Triton Compilation

triton-tvm

Triton to TVM transpiler.

nviwatch

NviWatch: A blazingly fast rust based TUI for managing and monitoring NVIDIA GPU processes

attention-gym

Helpful tools and examples for working with flex-attention

Stream-k.tvm

VideoSys

VideoSys: An easy and efficient system for video generation

ChatGLM-6B

ChatGLM-6B: An Open Bilingual Dialogue Language Model | 开源双语对话语言模型

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

JSQ

[ICML 2024] JSQ: Compressing Large Language Models by Joint Sparsification and Quantization

wanda

A simple and effective LLM pruning approach.

qlib

Qlib is an AI-oriented quantitative investment platform that aims to realize the potential, empower research, and create value using AI technologies in quantitative investment, from exploring ideas to implementing productions. Qlib supports diverse machine learning modeling paradigms. including supervised learning, market dynamics modeling, and RL.

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

TiledCUDA

TiledCUDA is a highly efficient kernel template library designed to elevate CUDA C’s level of abstraction for processing tiles.

Spec-Bench

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)

EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

veScale

A PyTorch Native LLM Training Framework

flute

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

mononn

SpeculativeDecodingPapers

📰 Must-read papers and blogs on Speculative Decoding ⚡️

specexec