sukjunhwang

ThunderKittens

Tile primitives for speedy kernels

llm.c

LLM training in simple, raw C/CUDA

ctm

unsloth

Finetune Llama 3, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory

torchtitan

A native PyTorch Library for large model training

llama3

The official Meta Llama 3 GitHub site

VAR

[GPT beats diffusion🔥] [scaling laws in visual generation📈] Official impl. of "Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction". An *ultra-simple, user-friendly yet state-of-the-art* codebase for autoregressive image generation!

attorch

A subset of PyTorch's neural network modules, written in Python using OpenAI's Triton.

lightning-thunder

Make PyTorch models up to 40% faster! Thunder is a source to source compiler for PyTorch. It enables using different hardware executors at once; across one or thousands of GPUs.

Score-Entropy-Discrete-Diffusion

[ICML 2024 Oral] Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution (https://arxiv.org/abs/2310.16834)

VISAGE

lectures

Material for cuda-mode lectures

resource-stream

CUDA related news and material links

sam

SAM: Sharpness-Aware Minimization (PyTorch)

LongMamba

Some preliminary explorations of Mamba's context scaling.

pytorch-lightning

Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.

SiT

Official PyTorch Implementation of "SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers"

maxdiffusion

contrastors

Train Models Contrastively in Pytorch

memray

Memray is a memory profiler for Python

cutlass

CUDA Templates for Linear Algebra Subroutines

visualwebarena

VisualWebArena is a benchmark for multimodal agents.

flash-linear-attention

Efficient implementations of state-of-the-art linear attention models in Pytorch and Triton

DCNv4

[CVPR 2024] Deformable Convolution v4

mvit

Code Release for MViTv2 on Image Recognition.

ml-cvnets

CVNets: A library for training computer vision networks

awesome-ssm-ml

Reading list for research topics in state-space models

LLM-Training-Puzzles

What would you do with 1000 H100s...

GPU-Puzzles

Solve puzzles. Learn CUDA.

Tensor-Puzzles

Solve puzzles. Improve your pytorch.