JiHa-Kim / hedgehog-porcupine-softmax-mimicry

Code implementation in PyTorch of the paper "The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry" at https://arxiv.org/abs/2402.04347

Hedgehog-Porcupine-Feature-Map

Code implementation in PyTorch of the paper "The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry"

Paper Link https://arxiv.org/abs/2402.04347

Appreciation

• Lucidrains
• Agora Research

Usage

Try running example.py

Architecture

Todo

License

Apache 2.0

Citations

@misc{zhang2024hedgehog, title={The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicry}, author={Michael Zhang and Kush Bhatia and Hermann Kumbong and Christopher Ré}, year={2024}, eprint={2402.04347}, archivePrefix={arXiv}, primaryClass={cs.LG} }