LittleQili

orion

An interference-aware scheduler for fine-grained GPU sharing

lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.

nccl-tests

NCCL Tests

Open-Sora

Open-Sora: Democratizing Efficient Video Production for All

FlexFlow

FlexFlow Serve: Low-Latency, High-Performance LLM Serving

llm-analysis

Latency and Memory Analysis of Transformer Models for Training and Inference

llm-awq

[MLSys 2024 Best Paper Award] AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

Fuser

A Fusion Code Generator for NVIDIA GPUs (commonly known as "nvFuser")

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

dlrover

DLRover: An Automatic Distributed Deep Learning System

distrifuser

[CVPR 2024 Highlight] DistriFusion: Distributed Parallel Inference for High-Resolution Diffusion Models

GPU-scheduler-for-deep-learning

GPU-scheduler-for-deep-learning

bamboo

Bamboo is a system for running large pipeline-parallel DNNs affordably, reliably, and efficiently using spot instances.

TGS

Artifacts for our NSDI'23 paper TGS

gemma.cpp

lightweight, standalone C++ inference engine for Google's Gemma models.

gemma_pytorch

The official PyTorch implementation of Google's Gemma models

gemma

Open weights LLM from Google DeepMind.

multi-gpu-programming-models

Examples demonstrating available options to program multiple GPUs in a single node or a cluster

PipeSwitch

PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications

glet

mig-parted

MIG Partition Editor for NVIDIA GPUs

MIGProfiler

Multi-Instance-GPU profiling tool

gdev

First-Class GPU Resource Management: Device Drivers, Runtimes, and CUDA Compilers for Nouveau.

flashinfer

FlashInfer: Kernel Library for LLM Serving

gdev

First-Class GPU Resource Management: Device Drivers, Runtimes, and CUDA Compilers for Nouveau.

slurm

Slurm: A Highly Scalable Workload Manager

TensorRT

NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference on NVIDIA GPUs. This repository contains the open source components of TensorRT.

pygmtools

A Python Graph Matching Toolkit.

nvidia-container-toolkit

Build and run containers leveraging NVIDIA GPUs

hidet

An open-source efficient deep learning framework/compiler, written in python.