llx-08

splitwise-sim

LLM serving cluster simulator

vattention

Dynamic Memory Management for Serving LLMs without PagedAttention

InfiniGen

InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management (OSDI'24)

FlashAttention-PyTorch

Implementation of FlashAttention in PyTorch

vidur

A large-scale simulation framework for LLM inference

KuiperLLama

动手实现大模型推理框架

ServerlessLLM

Cost-efficient and fast multi-LLM serving.

xFasterTransformer

ShiArthur03

Triton-Puzzles

Puzzles for learning Triton

how-to-optim-algorithm-in-cuda

how to optimize some algorithm in cuda.

FLASHNN

CUDA-Learn-Notes

🎉CUDA/C++ 笔记 / 大模型手撕CUDA / 技术博客，更新随缘: flash_attn、sgemm、sgemv、warp reduce、block reduce、dot product、elementwise、softmax、layernorm、rmsnorm、hist etc.

MInference

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

ncnn

ncnn is a high-performance neural network inference framework optimized for the mobile platform

googletest

GoogleTest - Google Testing and Mocking Framework

ring-flash-attention

Ring attention implementation with flash attention

freevpn

免费公益机场节点分享

ebooks

收藏的一些经典的历史、政治、心理、哲学、数学、计算机方面电子书(约10万本）

AI-Software-Startups

A Survey of AI startups

Mooncake

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.

ParrotServe

[OSDI'24] Serving LLM-based Applications Efficiently with Semantic Variable

CUDA_gemm

A simple high performance CUDA GEMM implementation.

lectures

Material for cuda-mode lectures

AI-System

System for AI Education Resource.

aiohttp

Asynchronous HTTP client/server framework for asyncio and Python

Awesome-RoadMaps-and-Interviews

Awesome Interviews for Coder, Programming Language, Software Engineering, Web, Backend, Distributed Infrastructure, DataScience & AI | 面试必备

cs344

Introduction to Parallel Programming class code

llumnix

Efficient and easy multi-instance LLM serving

Nsight-Compute-Docker-Image

Nsight Compute in Docker