BoruiXu

self-llm

《开源大模型食用指南》基于Linux环境快速部署开源大模型，更适合**宝宝的部署教程

word-GPT-Plus

Word GPT Plus is a word add-in which integrates the chatGPT model into Microsoft Word. Both official and web api is supported.

Chinese-LLaMA-Alpaca

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

mediapipe

Cross-platform, customizable ML solutions for live and streaming media.

qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

GEAR

GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM

calm

CUDA/Metal accelerated language model inference

fastllm

纯c++的全平台llm加速库，支持python调用，chatglm-6B级模型单卡可达10000+token / s，支持glm, llama, moss基座，手机端流畅运行

flash-llm

Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity

inferflow

Inferflow is an efficient and highly configurable inference engine for large language models (LLMs).

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

mlc-llm

Universal LLM Deployment Engine with ML Compilation

mixtral-offloading

Run Mixtral-8x7B models in Colab or consumer desktops

Edge-MoE

Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts

Anima

33B Chinese LLM, DPO QLORA, 100K context, AirLLM 70B inference with single 4GB GPU

LLM-Viewer

Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface.

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

intel-extension-for-transformers

⚡ Build your chatbot within minutes on your favorite device; offer SOTA compression techniques for LLMs; run LLMs efficiently on Intel Platforms⚡

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

DejaVu

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

FlexGen

Running large language models on a single GPU for throughput-oriented scenarios.

perf-book

The book "Performance Analysis and Tuning on Modern CPU"

llama3

The official Meta Llama 3 GitHub site

xdna-driver

Vitis-AI

Vitis AI is Xilinx’s development stack for AI inference on Xilinx hardware platforms, including both edge devices and Alveo cards.