lauthu

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

LLaVA

[NeurIPS'23 Oral] Visual Instruction Tuning (LLaVA) built towards GPT-4V level capabilities and beyond.

datasets

🤗 The largest hub of ready-to-use datasets for ML models with fast, easy-to-use and efficient data manipulation tools

nvidia-docker

Build and run Docker containers leveraging NVIDIA GPUs

LoRA

Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"

magic-animate

[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

micrograd

A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API

mistral-src

Reference implementation of Mistral AI 7B v0.1 model.

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

easy_rust

Rust explained using easy English

PowerInfer

High-speed Large Language Model Serving on PCs with Consumer-grade GPUs

skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

OLMo

Modeling, training, eval, and inference code for OLMo

distiller

Neural Network Distiller by Intel AI Lab: a Python package for neural network compression research. https://intellabs.github.io/distiller

exllamav2

A fast inference library for running LLMs locally on modern consumer-class GPUs

lightllm

LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.

Medusa

Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads

neural-compressor

SOTA low-bit LLM quantization (INT8/FP8/INT4/FP4/NF4) & sparsity; leading model compression techniques on TensorFlow, PyTorch, and ONNX Runtime

gptq

Code for the ICLR 2023 paper "GPTQ: Accurate Post-training Quantization of Generative Pretrained Transformers".

Olive

Olive: Simplify ML Model Finetuning, Conversion, Quantization, and Optimization for CPUs, GPUs and NPUs.

pytest-benchmark

py.test fixture for benchmarking code

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

onnxruntime-inference-examples

Examples for using ONNX Runtime for machine learning inferencing.

tensorrtllm_backend

The Triton TensorRT-LLM Backend

database-system-readings

:yum: A curated reading list about database systems

LLM-QAT

Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models"

transformer_vq

ppl.nn.llm

flash-attention

Fast and memory-efficient exact attention

Programming_Massively_Parallel_Processors

CUDA 6大并行计算模式 代码与笔记