This repository is a simple and clean GPT implementation in Tensorflow.
- Python 3.8
- TensorFow 2.8
- TensorFlow Text 2.8.1
- TensorFlow Datasets 4.8.1
- Datasets 2.8.0
The model trains by default on the OpenWebText dataset. Use --model_dir=<model_dir>
to provide the model directory name.
python train.py --model_dir=<model_dir>
Some other options:
- Use
--build_vocab=True
to build a WordPiece vocabulary.
Use --model_dir=<model_dir>
and --context=<context>
to provide the model directory name and context.
python generate.py --model_dir=<model_dir> --context=<context>
Adjust hyperparameters on the config.py
file.
Run tensorboard --logdir ./
.
- Improving language understanding by generative pre-training
- Language models are unsupervised multitask learners
- Language models are few-shot learners
- minGPT
Implementation notes:
- WordPiece tokenization
MIT