Weili17

ray-llm

RayLLM - LLMs on Ray

gpustack

Manage GPU clusters for running LLMs

giantpandacv.com

www.giantpandacv.com

ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

NeMo

A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speech)

AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

AutoGPTQ

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

meetups

sglang

SGLang is yet another fast serving framework for large language models and vision language models.

TransformerEngine

A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper and Ada GPUs, to provide better performance with lower memory utilization in both training and inference.

llama-cpp-python

Python bindings for llama.cpp

llama.cpp

llama 2 Inference

ollama

Get up and running with Llama 3.1, Mistral, Gemma 2, and other large language models.

step_into_llm

MindSpore online courses: Step into LLM

CUDA_Programming

《CUDA编程基础与实践》一书的代码

CUDA-Programming

Sample codes for my CUDA programming book

KuiperLLama

动手实现大模型推理框架

cuda-samples

Samples for CUDA Developers which demonstrates features in CUDA Toolkit

lightseq

LightSeq: A High Performance Library for Sequence Processing and Generation

Learn-CUDA-Programming

Learn CUDA Programming, published by Packt

lectures

Material for cuda-mode lectures

baby-llama2-chinese_cybertron

使用单个24G显卡，从0开始训练LLM

llama.cpp

LLM inference in C/C++

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

batch-prompting

[EMNLP 2023 Industry Track] A simple prompting approach that enables the LLMs to run inference in batches.

SpeculativeDecodingPapers

📰 Must-read papers and blogs on Speculative Decoding ⚡️

long-context-attention

Sequence Parallel Attention for Long Context LLM Model Training and Inference

SwiftSage

SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks

TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.