Jamie-Stirling / RetNet

An implementation of "Retentive Network: A Successor to Transformer for Large Language Models"

Jamie-Stirling/RetNet Issues

Is Retnet equivalent to ordinary GPT when the decay is set to 1 ?
Updated 5 months ago3
How to predict use this net?
Updated 7 months ago
a question about xpos and D of decay mat
Updated 7 months ago2
Dimensions of forward_recurrent
Closed 9 months ago5
Confusion about "the chunkwise recurrent representation of retention"
Updated 10 months ago
Can this mechanism be applied to PointCloud data ?
Updated 10 months ago
NO LM HEAD
Closed 10 months ago2
The complex theta should cancel out
Updated a year ago
/src/retnet.py GPU
Closed a year ago2
Assistance on training a new retention network model ?
Updated a year ago
Passing Attention Masks
Updated a year ago3
Error when the model is running on GPU
Updated a year ago1
Proposed improvement/collaboration: removing the O(T^2) training cost
Closed a year ago2
Q, k and D device difference
Closed a year ago1
can retnet be applied in point cloud tasks？
Updated a year ago
Changelog of official implementation
Updated a year ago2
Chunkwise retention giving different output
Closed a year ago4
what about cross-attention
Updated a year ago
demo example / number of parameter control vs original code
Updated a year ago4
Can you make this repo in available for package installers (pip)?
Updated a year ago
RetNet Officially Released
Closed a year ago1
Training is slow and some errors (perhaps)
Closed a year ago6
_get_D function very slow for long sequence
Closed a year ago1
Real-valued implementation using xPos
Closed a year ago
About the complex
Closed a year ago4
Some Questions about Attention Mask
Closed a year ago3