hanrui1sensetime

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

llmc

This is the official PyTorch implementation of "LLM-QBench: A Benchmark Towards the Best Practice for Post-training Quantization of Large Language Models", and also an efficient LLM compression tool with various advanced compression methods, supporting multiple inference backends.

llm_interview_note

主要记录大语言大模型（LLMs） 算法（应用）工程师相关的知识及面试题

qserve

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

Atom

[MLSys'24] Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

GPTQ-for-PULSE

4 bits quantization of PULSE models using GPTQ

QUIK

Repository for the QUIK project, enabling the use of 4bit kernels for generative inference

mamba

Mamba SSM architecture

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

RETFound_MAE

RETFound - A foundation model for retinal image

RETFound_MAE

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

awesome-lm-system

Summary of system papers/frameworks/codes/tools on training or serving large model

text-generation-inference

Large Language Model Text Generation Inference

ppl.llm.kernel.cuda

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

nndeploy

nndeploy是一款模型端到端部署框架。以多端推理以及基于有向无环图模型部署为基础，致力为用户提供跨平台、简单易用、高性能的模型部署体验。

mlc-llm

Universal LLM Deployment Engine with ML Compilation

OmniQuant

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

FlashAttention20Triton

Triton implementation of Flash Attention2.0

RPTQ-for-LLaMA

Efficient 3bit/4bit quantization of LLaMA models

FlexGen

Running large language models on a single GPU for throughput-oriented scenarios.

Awesome-Efficient-LLM

A curated list for Efficient Large Language Models

RPTQ4LLM

Reorder-based post-training quantization for large language model

PRIME

BROW

STU-Net

The largest pre-trained medical image segmentation model (1.4B parameters) based on the largest public dataset (>100k annotations) to date.

MIU-VL

This is a repository for the ICLR2023 accepted paper -- Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study.

CITE

[MICCAI'23] Text-guided Foundation Model Adaptation for Pathological Image Classification

XrayPULSE