linear-attention

There are 1 repository under linear-attention topic.

BlinkDL / RWKV-LM
RWKV (pronounced RwaKuv) is an RNN with great LLM performance, which can also be directly trained like a GPT transformer (parallelizable). We are at RWKV-7 "Goose". So it's combining the best of RNN and transformer - great performance, linear time, constant space (no kv-cache), fast training, infinite ctx_len, and free sentence embedding.
attention-mechanism chatgpt deep-learning gpt gpt-2 gpt-3 language-model linear-attention lstm pytorch rnn rwkv transformer transformers
Language:Python 14110
weigao266 / Awesome-Efficient-Arch
Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
linear-attention mamba sparse-attention state-space-model diffusion-llm efficient-architectures linear-models linear-rnn mixture-of-experts moe
355
happinesslz / LION
[NeurIPS 2024] Official code of ”LION: Linear Group RNN for 3D Object Detection in Point Clouds“
3d-object-detection linear-attention linear-rnn
Language:Python 209
lucidrains / taylor-series-linear-attention
Explorations into the recently proposed Taylor Series Linear Attention
artificial-intelligence attention-mechanisms deep-learning linear-attention
Language:Python 99
lucidrains / agent-attention-pytorch
Implementation of Agent Attention in Pytorch
artificial-intelligence attention-mechanisms deep-learning linear-attention
Language:Python 92
lironui / Multi-Attention-Network
The semantic segmentation of remote sensing images
attention-mechanism linear-attention remote-sensing segmentation semantic-segmentation
Language:Python 78
keshik6 / grafting
[NeurIPS 2025 Oral] Exploring Diffusion Transformer Designs via Grafting
architecture-editing architecture-research convolutions diffusion-models diffusion-transformer diffusion-transformers grafting hyena-operator hyena-x hyena-y image-generation linear-attention mamba-2 model-architecture-editing model-grafting post-training self-attention sub-quadratic-attention text-to-image-generation
Language:Jupyter Notebook 54
lironui / MAResU-Net
The semantic segmentation of remote sensing images
attention attention-mechanism linear-attention remote-sensing segmentation semantic-segmentation
Language:Python 50
lucidrains / autoregressive-linear-attention-cuda
CUDA implementation of autoregressive linear attention, with all the latest research findings
artificial-intelligence attention-mechanisms cuda deep-learning linear-attention
Language:Python 45
BICLab / MetaLA
Offical implementation of "MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map" (NeurIPS2024 Oral)
linear-attention llm
Language:Python 29
glassroom / heinsen_attention
Reference implementation of "Softmax Attention with Constant Cost per Token" (Heinsen, 2024)
attention attention-mechanism attention-model heinsen-attention linear-attention linear-attention-model
Language:Python 24
gmongaras / Cottention_Transformer
Code for the paper "Cottention: Linear Transformers With Cosine Attention"
attention linear-attention transformers
Language:Cuda 19
robflynnyh / hydra-linear-attention
Implementation of: Hydra Attention: Efficient Attention with Many Heads (https://arxiv.org/abs/2209.07484)
attention efficient-attention linear-attention machine-learning transformers
Language:Python 14
gmlwns2000 / sea-attention
Official Implementation of SEA: Sparse Linear Attention with Estimated Attention Mask (ICLR 2024)
attention efficient-attention linear-attention sea-attention
Language:Python 10
OSU-STARLAB / LeaPformer
[ICML 2024] Official implementation of "LeaPformer: Enabling Linear Transformers for Autoregressive and Simultaneous Tasks via Learned Proportions."
efficiency language-modeling linear-attention long-range-arena simultaneous-translation transformer-architecture
Language:Python 10
RWKV-Wiki / rwkv-wiki.github.io
RWKV Wiki website (archived, please visit official wiki)
attention-mechanism deep-learning gpt gpt-2 gpt-3 language-model linear-attention lstm rnn rwkv transformer transformers
Language:Shell 10
mtanghu / LEAP
LEAP: Linear Explainable Attention in Parallel for causal language modeling with O(1) path length, and O(1) inference
linear-attention pytorch transformers attention-mechanism deep-learning local-attention softmax additive-attention dot-product-attention parallel rnn transformer
Language:Jupyter Notebook 4
vrund038 / lm
🔍 Enhance your workflow with Houtini LM, an MCP server that offloads code analysis and documentation tasks to LM Studio, streamlining your development process.
attention-mechanism chatgpt chinese daw deep-learning evaluation-framework gerrit-mirror gpt hacktoberfest linear-attention model-para pretrained-models python rlhf rwkv transformer transformers typescript
Language:TypeScript 3
tzhengtek / saute
SAUTE is a lightweight transformer-based architecture adapted for dialog modeling
linear-attention machine-learning masked-language-models speaker-embedding transformers tsinghua-university
Language:Python 2
Rushi314 / Transformers-for-high-resolution-image-synthesis
Taming Transformers for High-Resolution Image Synthesis
computer-vision gans linear-attention transformers-gpt2 sflickr
Language:Jupyter Notebook 0
HassanJbara / lin-attn-bench
Independent and reproducable benchmarking of linear attention models
attention benchmarking evaluation linear-attention
Language:Python
HassanJbara / lin-attn-zoo
Pure PyTorch implementations of Popular Linear Attention models
large-language-models linear-attention pytorch rnn
Language:Python

linear-attention

BlinkDL / RWKV-LM

weigao266 / Awesome-Efficient-Arch

happinesslz / LION

lucidrains / taylor-series-linear-attention

lucidrains / agent-attention-pytorch

lironui / Multi-Attention-Network

keshik6 / grafting

lironui / MAResU-Net

lucidrains / autoregressive-linear-attention-cuda

BICLab / MetaLA

glassroom / heinsen_attention

gmongaras / Cottention_Transformer

robflynnyh / hydra-linear-attention

gmlwns2000 / sea-attention

OSU-STARLAB / LeaPformer

RWKV-Wiki / rwkv-wiki.github.io

mtanghu / LEAP

vrund038 / lm

tzhengtek / saute

Rushi314 / Transformers-for-high-resolution-image-synthesis

HassanJbara / lin-attn-bench

HassanJbara / lin-attn-zoo