efficient-attention

There are 1 repository under efficient-attention topic.

thu-ml / SageAttention
[ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves speedup of 2-5x compared to FlashAttention, without losing end-to-end metrics across language, image, and video models.
attention cuda efficient-attention inference-acceleration llm llm-infra mlsys quantization triton video-generate video-generation vit
Language:Cuda 2634
lucidrains / ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
attention-mechanism efficient-attention long-context distributed-attention
Language:Python 540
lucidrains / CoLT5-attention
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
artificial-intelligence attention-mechanisms deep-learning efficient-attention routing
Language:Python 230
jlamprou / Infini-Attention
Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval
attention efficient-attention infinite llm qwen transformer
Language:Python 84
Ascend-Research / CascadedGaze
The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.
deblurring denoising efficiency efficient-attention image-restoration transformer
Language:Python 77
davidsvy / cosformer-pytorch
Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".
neural-network transformer pytorch attention-mechanism deep-learning artificial-intelligence efficient-attention iclr2022 iclr
Language:Jupyter Notebook 44
HolmesShuan / Compact-Global-Descriptor
Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).
attention-mechanism attention-model convolutional-neural-networks efficient efficient-attention pytorch
Language:Python 25
robflynnyh / hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
attention efficient-attention linear-attention machine-learning transformers
Language:Python 14
gmlwns2000 / sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
attention efficient-attention linear-attention sea-attention
Language:Python 11
MAGICS-LAB / NonparametricHopfield
Nonparametric Modern Hopfield Models
efficient-attention efficient-hopfield-models efficient-hopfield-networks efficient-transformers modern-hopfield-model modern-hopfield-networks
Language:Jupyter Notebook 8
priyanshujiiii / awesome-Attention
Resources and references on solved and unsolved problems in attention mechanisms.
attention attention-mechanism llm ai-research deep-learning efficient-attention machine-learning solved-problems survey-paper transformer unsolved-problems
pszemraj / samba-pytorch
Minimal implementation of Samba by Microsoft in PyTorch
efficient-attention language-model llm long-context-modeling mamba-state-space-models pytorch-implementation ssm
Language:Python

efficient-attention

thu-ml / SageAttention

lucidrains / ring-attention-pytorch

lucidrains / CoLT5-attention

jlamprou / Infini-Attention

Ascend-Research / CascadedGaze

davidsvy / cosformer-pytorch

HolmesShuan / Compact-Global-Descriptor

robflynnyh / hydra-linear-attention

gmlwns2000 / sea-attention

MAGICS-LAB / NonparametricHopfield

priyanshujiiii / awesome-Attention

pszemraj / samba-pytorch