shreyansh26 / resource-stream

CUDA related news and material links

CUDA MODE Resource Stream

Here you find a collection of CUDA related material (books, papers, blog-post, youtube videos, tweets, implementations etc.). We also collect information to higher level tools for performance optimization and kernel development like Triton and torch.compile() ... whatever makes the GPUs go brrrr.

You know a great resource we should add? Please see How to contribute.

1st Contact with CUDA

• An Easy Introduction to CUDA C and C++
• CUDA Toolkit Documentation
• Basic terminology: Thread block, Warp, Streaming Multiprocessor: Wiki: Thread Block, A tour of CUDA

Papers, Case Studies

• A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library

Books

• Programming Massively Parallel Processors: A Hands-on Approach

Tri Dao Fan Section

• Dao-AILab/flash-attention
• state-spaces/mamba

Practice

• Sasha Rush's GPU Puzzles

PyTorch Highlights

• Accelerating Generative AI with PyTorch: Segment Anything, Fast
• Accelerating Generative AI with PyTorch II: GPT, Fast
• PyTorch Compiler Troubleshooting

Code / Libs

• NVIDIA/cutlass

Essentials

• Triton compiler tutorials
• CUDA C++ Programming Guide
• pybind11 documentation

News

• SemiAnalysis

Hardware Architecture

• NVIDIA H100 Whitepaper
• NVIDIA GH200 Whitepaper
• AMD CDNA 2 (MI200) Whitepaper
• AMD MI300X Data Sheet

How to contribute

To share interesting CUDA related links please create a pull request for this file. See editing files in the github documentation.

Or contact us on the CUDA MODE discord server: invitation link TBD