The code in this repository accompanies the research paper, What Algorithms can Transformers Learn? A Study in Length Generalization. It is provided for reference and research purposes only.
We include the NumPy-based implementation of RASP-L, as well as several of the code listings from the above paper. We also suggest taking a look at raskell, an independent implementation of RASP-L in Haskell.
Clone the repo, then install with:
pip install -e .
To import all RASP-L core and library functions:
from np_rasp import *
For example, see add.py for RASP-L programs for forward and reverse addition. You can run several addition tests via:
cd examples
python test_add.py
@misc{zhou2023algorithms,
title={What Algorithms can Transformers Learn? A Study in Length Generalization},
author={Hattie Zhou and Arwen Bradley and Etai Littwin and Noam Razin and Omid Saremi and Josh Susskind and Samy Bengio and Preetum Nakkiran},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://arxiv.org/abs/2310.16028}
}