lzzmm

llm_aided_ocr

Enhance Tesseract OCR output for scanned PDFs by applying Large Language Model (LLM) corrections.

TensorRT-Incubator

Experimental projects related to TensorRT

chatbox

User-friendly Desktop Client App for AI Models/LLMs (GPT, Claude, Gemini, Ollama...)

flash-attention-minimal

Flash Attention in ~100 lines of CUDA (forward pass only)

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

MiniCPM-V

MiniCPM-V 2.6: A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone

gpu.cpp

A lightweight library for portable low-level GPU computation using WebGPU.

turingas

Assembler for NVIDIA Volta and Turing GPUs

MInference

To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.

sarathi-serve

A low-latency & high-throughput serving engine for LLMs

fp6_llm

An efficient GPU support for LLM inference with x-bit quantization (e.g. FP6,FP5).

perf-ninja

This is an online course where you can learn and master the skill of low-level performance analysis and tuning.

BurstGPT

A ChatGPT(GPT-3.5) & GPT-4 Workload Trace to Optimize LLM Serving Systems

awesome-local-ai

An awesome repository of local AI tools

cortex

Drop-in, local AI alternative to the OpenAI stack. Multi-engine (llama.cpp, TensorRT-LLM, ONNX). Powers 👋 Jan

LLaMA-Factory

A WebUI for Efficient Fine-Tuning of 100+ LLMs (ACL 2024)

cppinsights

C++ Insights - See your source code with the eyes of a compiler

Pruner-Zero

Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for LLMs

llama3

The official Meta Llama 3 GitHub site

compiler-and-arch

A list of tutorials, paper, talks, and open-source projects for emerging compiler and architecture

llama3-from-scratch

llama3 implementation one matrix multiplication at a time

vidur

A large-scale simulation framework for LLM inference

HEBO

Bayesian optimisation & Reinforcement Learning library developped by Huawei Noah's Ark Lab

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

llm-benchmark

llm.c

LLM training in simple, raw C/CUDA

triton

Development repository for the Triton language and compiler

apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

tvm

Open deep learning compiler stack for cpu, gpu and specialized accelerators

mlc-llm

Universal LLM Deployment Engine with ML Compilation