zzpmiracle

vattention

Dynamic Memory Management for Serving LLMs without PagedAttention

ScaleLLM

A high-performance inference system for large language models, designed for production environments.

FLASHNN

DistServe

Disaggregated serving system for Large Language Models (LLMs).

llumnix

Efficient and easy multi-instance LLM serving

Spec-Bench

Spec-Bench: A Comprehensive Benchmark and Unified Evaluation Platform for Speculative Decoding (ACL 2024 Findings)

DARC

Decentralized Autonomous Regulated Company (DARC), a company virtual machine that runs on any EVM-compatible blockchain, with on-chain law system, multi-level tokens and dividends mechanism.

FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

flashinfer

FlashInfer: Kernel Library for LLM Serving

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

Modern-CPP-Programming

Modern C++ Programming Course (C++03/11/14/17/20/23/26)

rtp-llm

RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.

how-to-optim-algorithm-in-cuda

how to optimize some algorithm in cuda.

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

ChatGLM3

ChatGLM3 series: Open Bilingual Chat LLMs | 开源双语对话语言模型

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

recom

An Optimizing Compiler for Recommendation Model Inference

sd-webui-EasyPhoto

📷 EasyPhoto | Your Smart AI Photo Generator.

diffusers-api

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

hiq

HiQ - Observability And Optimization In Modern AI Era

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

NeMo-Framework-Launcher

Provides end-to-end model development pipelines for LLMs and Multimodal models that can be launched on-prem or cloud-native.

YesPlayMusic

高颜值的第三方网易云播放器，支持 Windows / macOS / Linux :electron:

BMTrain

Efficient Training (including pre-training and fine-tuning) for Big Models

Grounded-Segment-Anything

Grounded SAM: Marrying Grounding DINO with Segment Anything & Stable Diffusion & Recognize Anything - Automatically Detect , Segment and Generate Anything

the-algorithm

Source code for Twitter's Recommendation Algorithm

ChatGPT-Next-Web

A cross-platform ChatGPT/Gemini UI (Web / PWA / Linux / Win / MacOS). 一键拥有你自己的跨平台 ChatGPT/Gemini 应用。

Pake

🤱🏻 Turn any webpage into a desktop app with Rust. 🤱🏻 利用 Rust 轻松构建轻量级多端桌面应用