DouglasOrr / BBLM

Bare bones language modelling, in PyTorch

Bare Bones Language Model

A PyTorch starter for doing something actually interesting.

Development

This project follows these principles:

• Explicit, because you shouldn't have to guess
• Dependency-light, because dependencies break
• Concise, because verbose code is skimmed & ignored

Setup

python3 -m venv .venv
# Add to the end of .venv/bin/activate
# export PYTHONPATH="${PYTHONPATH}:$(dirname ${VIRTUAL_ENV})"

source .venv/bin/activate
pip install wheel
# pip install torch --index-url https://download.pytorch.org/whl/cpu   # for CPU
pip install -r requirements-dev.txt

./dev  # run CI checks

References

• WikiText-103 dataset, Pointer Sentinel Mixture Models (2016), Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher.
• Transformer, Attention Is All You Need (2017), Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin.