Collection of effcient implementations of attention with CUDA.
These two papers describe how to implement Exact attention while avoid O(n^2) memory occupied by intermediate attenion matrix (i.e. QK^T).
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- Self-attention Does Not Need O(n2) Memory
Implementations can be found at:
Sparse GEMMs are not friendly with GPUs due to poor spatial-temporal locality. But stuctured block sparsity solves this problem. See OpenAI's blog for details.