Here you find a collection of CUDA related material (books, papers, blog-post, youtube videos, tweets, implementations etc.). We also collect information to higher level tools for performance optimization and kernel development like Triton and torch.compile() ... whatever makes the GPUs go brrrr.
You know a great resource we should add? Please see How to contribute.
- An Easy Introduction to CUDA C and C++
- CUDA Toolkit Documentation
- Basic terminology: Thread block, Warp, Streaming Multiprocessor: Wiki: Thread Block, A tour of CUDA
- GPU Performance Background User's Guide
- OLCF NVIDIA CUDA Training Series, talk recordings can be found under the presentation footer for each lecture; exercises
- GTC 2022 - CUDA: New Features and Beyond - Stephen Jones
- Intro video: Writing Code That Runs FAST on a GPU
- A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library
- How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog
- Anatomy of high-performance matrix multiplication
- Programming Massively Parallel Processors: A Hands-on Approach
- Cuda by Example: An Introduction to General-Purpose Gpu Programming; code
- The CUDA Handbook
- x: @tri_dao
- Dao-AILab/flash-attention, paper
- state-spaces/mamba, paper: Mamba: Linear-Time Sequence Modeling with Selective State Spaces, minimal impl: mamba-minimal
- Accelerating Generative AI with PyTorch: Segment Anything, Fast
- Accelerating Generative AI with PyTorch II: GPT, Fast
- Speed, Python: Pick Two. How CUDA Graphs Enable Fast Python Code for Deep Learning
- Performance Debugging of Production PyTorch Models at Meta
- PyTorch Compiler Troubleshooting
- PyTorch internals
- Pytorch 2 internals
- Understanding GPU memory: 1: Visualizing All Allocations over Time, 2: Finding and Removing Reference Cycles
- Debugging memory using snapshots: Debugging PyTorch memory use with snapshots
- Trace Analyzer: PyTorch Trace Analysis for the Masses
- Triton compiler tutorials
- CUDA C++ Programming Guide
- PyTorch: Custom C++ and CUDA Extensions, Code: pytorch/extension-cpp
- PyTorch C++ API
- pybind11 documentation
- NVIDIA Tensor Core Programming
- GPU Programming: When, Why and How?
- How GPU Computing Works | GTC 2021 (more basic than the 2022 version)
- How CUDA Programming Works | GTC 2022
- Nsight Compute Profiling Guide
- mcarilli/nsight.sh - Favorite nsight systems profiling commands for PyTorch scripts
- Profiling GPU Applications with Nsight Systems
- PyTorch
- Trtion, github: openai/triton
- numba @cuda.jit
- Apache TVM
- JAX Pallas
- CuPy NumPy compatible GPU Computing
- Codon @gpu.kernel, github: exaloop/codon
- Mojo (part of commercial MAX Plattform by Modular)
- NVIDIA Python Bindings: CUDA Python (calling NVRTC to compile kernels, malloc, copy, launching kernels, ..), cuDNN FrontEnd(FE) API, CUTLASS Python Interface
- GPU Gems: Parallel Prefix Sum (Scan) with CUDA, PDF version (2007), impl: stack overflow, nicer impl: mattdean1/cuda
- Accelerating Reduction and Scan Using Tensor Core Units
- Thrust: Prefix Sums, Reference: scan variants
- CUB, part of cccl: NVIDIA/cccl/tree/main/cub
- SAM Algorithm: Higher-Order and Tuple-Based Massively-Parallel Prefix Sums (licensed for non commercial use only)
- CUB Algorithm: Single-pass Parallel Prefix Scan with Decoupled Look-back
- Group Experiments: johnryan465/pscan, andreaskoepf/pscan_kernel
unsloththat implements custom kernels in Triton for faster QLoRA training- Custom implementation of relative position attention (link)
- Tri Dao's Triton implementation of Flash Attention: flash_attn_triton.py
To share interesting CUDA related links please create a pull request for this file. See editing files in the github documentation.
Or contact us on the CUDA MODE discord server: https://discord.gg/XsdDHGtk9N