reppertj

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

guidance

A guidance language for controlling large language models.

mlc-llm

Universal LLM Deployment Engine with ML Compilation

dspy

DSPy: The framework for programming—not prompting—foundation models

litgpt

20+ high-performance LLMs with recipes to pretrain, finetune and deploy at scale.

OpenChatKit

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

axolotl

Go ahead and axolotl questions

skypilot

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

xtuner

An efficient, flexible and full-featured toolkit for fine-tuning LLM (InternLM2, Llama3, Phi3, Qwen, Mistral, ...)

CTranslate2

Fast inference engine for Transformer models

datasketch

MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW

datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

schedule_free

Schedule-Free Optimization in PyTorch

nanotron

Minimalistic large language model 3D-parallelism training

pyvene

Stanford NLP Python Library for Understanding and Improving PyTorch Models via Interventions

FLARE

Forward-Looking Active REtrieval-augmented generation (FLARE)

Long-Context

This repository contains code and tooling for the Abacus.AI LLM Context Expansion project. Also included are evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabilities with context expansion. We also include key experimental results and instructions for reproducing and building on them.

ranx

⚡️A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion 🐍

torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

clownfish

Constrained Decoding for LLMs against JSON Schema

tensorizer

Module, Model, and Tensor Serialization/Deserialization

magix

Supercharge huggingface transformers with model parallelism.

scirepeval

SciRepEval benchmark training and evaluation scripts

nccl-tests

NVIDIA NCCL Tests for Distributed Training

learned-sparse-retrieval

Unified Learned Sparse Retrieval Framework

qdrant-lib

Extract core logic from qdrant and make it available as a library.

ml-containers

spark-on-k8s-images

Driver/Executor images for spark-operator