lvhan028

langchain

🦜🔗 Build context-aware reasoning applications

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

mlc-llm

Universal LLM Deployment Engine with ML Compilation

Chinese-LLaMA-Alpaca

中文LLaMA&Alpaca大语言模型+本地CPU/GPU训练部署 (Chinese LLaMA & Alpaca LLMs)

chatgpt-mirai-qq-bot

🚀 一键部署！真正的 AI 聊天机器人！支持ChatGPT、文心一言、讯飞星火、Bing、Bard、ChatGLM、POE，多账号，人设调教，虚拟女仆、图片渲染、语音发送 | 支持 QQ、Telegram、Discord、微信 等平台

mamba

Mamba SSM architecture

tiktoken

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

MOSS

An open-source tool-augmented conversational language model from Fudan University

mistral-src

Reference implementation of Mistral AI 7B v0.1 model.

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

streaming-llm

[ICLR 2024] Efficient Streaming Language Models with Attention Sinks

InternLM

Official release of InternLM2.5 base and chat models. 1M context support

sglang

SGLang is a fast serving framework for large language models and vision language models.

ToolBench

[ICLR'24 spotlight] An open platform for training, serving, and evaluating large language model for tool learning.

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

opencompass

OpenCompass is an LLM evaluation platform, supporting a wide range of models (Llama3, Mistral, InternLM2,GPT-4,LLaMa2, Qwen,GLM, Claude, etc) over 100+ datasets.

xtuner

An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)

fastllm

纯c++的全平台llm加速库，支持python调用，chatglm-6B级模型单卡可达10000+token / s，支持glm, llama, moss基座，手机端流畅运行

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

lagent

A lightweight framework for building LLM-based agents

HuixiangDou

HuixiangDou: Overcoming Group Chat Scenarios with LLM-based Technical Assistance

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

torchsparse

[MICRO'23, MLSys'22] TorchSparse: Efficient Training and Inference Framework for Sparse Convolution on GPUs.

GPT4RoI

GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest

OpenAOE

LLM Group Chat Framework: chat with multiple LLMs at the same time. 大模型群聊框架：同时与多个大语言模型聊天。

INT8-Flash-Attention-FMHA-Quantization