cerebellumking

the-art-of-command-line

Master the command line, in one page

mac-setup

Installing Development environment on macOS

llm-action

本项目旨在分享大模型相关技术原理以及实战经验。

speculative-decoding

Explorations into some recent techniques surrounding speculative decoding

MediaCrawler

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

lnav

Log file navigator

LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

Awesome-LLMs-on-device

Awesome LLMs on Device: A Comprehensive Survey

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

text-generation-inference

Large Language Model Text Generation Inference

specinfer-ae

BigLittleDecoder

[NeurIPS'23] Speculative Decoding with Big Little Decoder

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

REST

REST: Retrieval-Based Speculative Decoding, NAACL 2024

LookaheadDecoding

[ICML 2024] Break the Sequential Dependency of LLM Inference Using Lookahead Decoding

Spec-Bench

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)

SpeculativeDecodingPapers

📰 Must-read papers and blogs on Speculative Decoding ⚡️

fastkron

LLMCompass

flute

Fast Matrix Multiplications for Lookup Table-Quantized LLMs

keyformer-llm

prompt-cache

Modular and structured prompt caching for low-latency LLM inference

llumnix

Efficient and easy multi-instance LLM serving

inference

Reference implementations of MLPerf™ inference benchmarks

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

AutoGPTQ

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

QQQ

QQQ is an innovative and hardware-optimized W4A8 quantization solution for LLMs.