char-1ee

Mooncake

OpenDiT

OpenDiT: An Easy, Fast and Memory-Efficient System for DiT Training and Inference

cudf

cuDF - GPU DataFrame Library

duckdb

DuckDB is an analytical in-process SQL database management system

Serving

A flexible, high-performance carrier for machine learning models（『飞桨』服务化部署框架）

desktop

Replit Desktop App

tensorrtllm_backend

The Triton TensorRT-LLM Backend

paddler

Stateful load balancer custom-tailored for llama.cpp

NPKit

NCCL Profiling Kit

dlrover

DLRover: An Automatic Distributed Deep Learning System

DistServe

Disaggregated serving system for Large Language Models (LLMs).

mindcraft

inference

Reference implementations of MLPerf™ inference benchmarks

ThunderKittens

Tile primitives for speedy kernels

AFFiNE

There can be more than Notion and Miro. AFFiNE(pronounced [ə‘fain]) is a next-gen knowledge base that brings planning, sorting and creating all together. Privacy first, open-source, customizable and ready to use.

core

MoonBit's Core library

DeepSeek-V2

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model

Awesome-LLM-Inference

📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.

llm-scheduling-artifact

Artifact of OSDI '24 paper, ”Llumnix: Dynamic Scheduling for Large Language Model Serving“

transformer-debugger

llama3

The official Meta Llama 3 GitHub site

experiments

My exploration on new technologies.

attorch

A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.

xformers

Hackable and optimized Transformers building blocks, supporting a composable construction.

AISystem

AISystem 主要是指AI系统，包括AI芯片、AI编译器、AI推理和训练框架等AI全栈底层技术

triton

Development repository for the Triton language and compiler

attention_with_linear_biases

Code for the ALiBi method for transformer language models (ICLR 2022)

llm.c

LLM training in simple, raw C/CUDA

public-apis

A collective list of free APIs

mistral-inference

Official inference library for Mistral models