efficient-attention

There are 1 repository under efficient-attention topic.

thu-ml / SageAttention
Quantized Attention achieves speedup of 2-5x and 3-11x compared to FlashAttention and xformers, without lossing end-to-end metrics across language, image, and video models.
attention inference-acceleration llm quantization cuda triton video-generation efficient-attention mlsys llm-infra vit video-generate
Language:Cuda 2390
lucidrains / ring-attention-pytorch
Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
attention-mechanism efficient-attention long-context distributed-attention
Language:Python 510
lucidrains / CoLT5-attention
Implementation of the conditionally routed attention in the CoLT5 architecture, in Pytorch
artificial-intelligence attention-mechanisms deep-learning efficient-attention routing
Language:Python 226
jlamprou / Infini-Attention
Efficient Infinite Context Transformers with Infini-attention Pytorch Implementation + QwenMoE Implementation + Training Script + 1M context keypass retrieval
attention efficient-attention infinite llm transformer qwen
Language:Python 81
Ascend-Research / CascadedGaze
The official PyTorch implementation for CascadedGaze: Efficiency in Global Context Extraction for Image Restoration, TMLR'24.
deblurring denoising efficiency image-restoration transformer efficient-attention
Language:Python 63
davidsvy / cosformer-pytorch
Unofficial PyTorch implementation of the paper "cosFormer: Rethinking Softmax In Attention".
neural-network transformer pytorch attention-mechanism deep-learning artificial-intelligence efficient-attention iclr2022 iclr
Language:Jupyter Notebook 44
HolmesShuan / Compact-Global-Descriptor
Pytorch implementation of "Compact Global Descriptor for Neural Networks" (CGD).
attention-mechanism convolutional-neural-networks pytorch efficient efficient-attention attention-model
Language:Python 25
robflynnyh / hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
attention efficient-attention linear-attention machine-learning transformers
Language:Python 14
gmlwns2000 / sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
attention efficient-attention linear-attention sea-attention
Language:Python 10
MAGICS-LAB / NonparametricHopfield
Nonparametric Modern Hopfield Models
efficient-attention efficient-transformers modern-hopfield-model modern-hopfield-networks efficient-hopfield-models efficient-hopfield-networks
Language:Jupyter Notebook 7
pszemraj / samba-pytorch
Minimal implementation of Samba by Microsoft in PyTorch
efficient-attention language-model llm long-context-modeling ssm mamba-state-space-models pytorch-implementation
Language:Python

efficient-attention

thu-ml / SageAttention

lucidrains / ring-attention-pytorch

lucidrains / CoLT5-attention

jlamprou / Infini-Attention

Ascend-Research / CascadedGaze

davidsvy / cosformer-pytorch

HolmesShuan / Compact-Global-Descriptor

robflynnyh / hydra-linear-attention

gmlwns2000 / sea-attention

MAGICS-LAB / NonparametricHopfield

pszemraj / samba-pytorch