beimingxinghai

pytest-xprocess

pytest external process plugin

sglang

SGLang is yet another fast serving framework for large language models and vision language models.

mem0

The memory layer for Personalized AI

LookaheadDecoding

RouteLLM

A framework for serving and evaluating LLM routers - save LLM costs without compromising quality!

circuitbreaker

Python "Circuit Breaker" implementation

pybreaker

Python implementation of the Circuit Breaker pattern.

punica

Serving multiple LoRA finetuned LLM as one

Docker_training_with_DockerMe

The tools and sample needed to learn the Docker

nvitop

An interactive NVIDIA-GPU process viewer and beyond, the one-stop solution for GPU process management.

dcgm-exporter

NVIDIA GPU metrics exporter for Prometheus leveraging DCGM

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

AutoAWQ_kernels

python_ebook

收集了一些Python相关资料

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

flashinfer

FlashInfer: Kernel Library for LLM Serving

marlin

FP16xINT4 LLM inference kernel that can achieve near-ideal ~4x speedups up to medium batchsizes of 16-32 tokens.

kiss-translator

A simple, open source bilingual translation extension & Greasemonkey script (一个简约、开源的 双语对照翻译扩展 & 油猴脚本)

kernel_tuner

Kernel Tuner

pytest-xdist

pytest plugin for distributed testing and loop-on-failures testing modes.

DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

CSView

CSView是一个互联网面试知识学习和汇总项目，包括面试高频算法、系统设计、计算机网络、操作系统、C++、Java、golang、MySQL、Redis、K8s、消息队列等常见面试题。

k8s-books

2021年最新linux运维面试题，k8s面试题，kubernetes面试题，Linux运维面试题，K8s视频教程，Docker面试题，kubernetes视频，等资料收集分享

k8s_awesome_document

【2021年新鲜出炉】K8s（Kubernetes）的工程师资料合辑，书籍推荐，面试题，精选文章，开源项目，PPT，视频，大厂资料

llmperf

LLMPerf is a library for validating and benchmarking LLMs

llmperf-leaderboard

LLMSpeculativeSampling

Fast inference from large lauguage models via speculative decoding

exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

mlc-llm

Universal LLM Deployment Engine with ML Compilation

lsd

The next gen ls command