jiashu-z

SpotServe

SpotServe: Serving Generative Large Language Models on Preemptible Instances

readerwriterqueue

A fast single-producer, single-consumer lock-free queue for C++

lorax

Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs

transformer-walkthrough

A walkthrough of transformer architecture code

bam

intel-extension-for-deepspeed

Intel® Extension for DeepSpeed* is an extension to DeepSpeed that brings feature support with SYCL kernels on Intel GPU(XPU) device. Note XPU is already supported by stock DeepSpeed.

generative-ai-for-beginners

18 Lessons, Get Started Building with Generative AI 🔗 https://microsoft.github.io/generative-ai-for-beginners/

DALI

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

dlrm_datasets

Set of datasets for the deep learning recommendation model (DLRM).

sc23-dl-tutorial

SC23 Deep Learning at Scale Tutorial Material

eecs598

Advanced Topics on Systems for X

ppl.llm.kernel.cuda

llm-analysis

Latency and Memory Analysis of Transformer Models for Training and Inference

punica

Serving multiple LoRA finetuned LLM as one

scalene

Scalene: a high-performance, high-precision CPU, GPU, and memory profiler for Python with AI-powered optimization proposals

qdrant

Qdrant - High-performance, massive-scale Vector Database for the next generation of AI. Also available in the cloud https://cloud.qdrant.io/

llmperf

LLMPerf is a library for validating and benchmarking LLMs

DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

kineto

A CPU+GPU Profiling library that provides access to timeline traces and hardware performance counters.

how-to-optimize-gemm

row-major matmul optimization

minGPT

A minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training

mindsdb

The platform for building AI from enterprise data

Megatron-LM

Ongoing research training transformer models at scale

megablocks

CUDA_Freshman

ElasticFlow

Artifacts for our ASPLOS'23 paper ElasticFlow

gpu_poor

Calculate token/s & GPU memory requirement for any LLM. Supports llama.cpp/ggml/bnb/QLoRA quantization

EnvPipe

FastCkpt

Python package for rematerialization-aware gradient checkpointing

ssb-dbgen