zzxxxl

MobileAgent

Mobile-Agent: The Powerful Mobile Device Operation Assistant Family

refusal_direction

Code and results accompanying the paper "Refusal in Language Models Is Mediated by a Single Direction".

einops

Flexible and powerful tensor operations for readable and reliable code (for pytorch, jax, TF and others)

mllm

Fast Multimodal LLM on Mobile Devices

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

mnn-llm

llm deploy project based mnn.

maid

Maid is a cross-platform Flutter app for interfacing with GGUF / llama.cpp models locally, and with Ollama and OpenAI models remotely.

InferLLM

a lightweight LLM model inference framework

llm-applications

A comprehensive guide to building RAG-based LLM applications for production.

TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library

mlc-llm

Universal LLM Deployment Engine with ML Compilation

MergeLM

Codebase for Merging Language Models (ICML 2024)

llama3-from-scratch

llama3 implementation one matrix multiplication at a time

micrograd

A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API

KsanaLLM

fastllm

纯c++的全平台llm加速库，支持python调用，chatglm-6B级模型单卡可达10000+token / s，支持glm, llama, moss基座，手机端流畅运行

MiniCPM

MiniCPM3-4B: An edge-side LLM that surpasses GPT-3.5-Turbo.

Qwen2.5

Qwen2.5 is the large language model series developed by Qwen team, Alibaba Cloud.

sglang

SGLang is a fast serving framework for large language models and vision language models.

DejaVu

Awesome-LLM-System-Papers

dpo_toxic

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity.

ollama

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.

MambaOut

MambaOut: Do We Really Need Mamba for Vision?

FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

gptq

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

gpt-fast

Simple and efficient pytorch-native transformer text generation in <1000 LOC of python.

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.