berlino / seq-test

Understand and test language model architectures on synthetic tasks.

Setup

pip install torch==2.2.0 opt_einsum pythomata
pip install -U git+https://github.com/sustcsonglin/flash-linear-attention
pip install -e .

Running

python -m zoology.launch zoology/experiments/mqar/mha.py

Sync with Upstream

git checkout -b HazyResearch-main main
git pull https://github.com/HazyResearch/zoology.git main
git checkout main
git merge --no-ff HazyResearch-main
git push origin main