awf / functional-transformer

A pure-functional implementation of a machine learning transformer model in Python/JAX

awf/functional-transformer Issues

Make value heads nonsquare and add back head concatenation
Updated 2 years ago
Add non-learned position encodings
Updated 2 years ago
Initialization of positional encodings?
Closed 2 years ago3
Bug? Loss curves show a distinct correlation with batch id.
Updated 2 years ago