TushaarGVS / GPT

Implementation of Generative Pre-trained Transformer Model in Tensorflow

GPT

This repository is a simple and clean GPT implementation in Tensorflow.

Dependencies

• Python 3.8
• TensorFow 2.8
• TensorFlow Text 2.8.1
• TensorFlow Datasets 4.8.1
• Datasets 2.8.0

Usage

Train

The model trains by default on the OpenWebText dataset. Use --model_dir=<model_dir> to provide the model directory name.

python train.py --model_dir=<model_dir> 

Some other options:

• Use --build_vocab=True to build a WordPiece vocabulary.

Generate

Use --model_dir=<model_dir> and --context=<context> to provide the model directory name and context.

python generate.py --model_dir=<model_dir> --context=<context>

Hparams setting

Adjust hyperparameters on the config.py file.

Tensorboard

Run tensorboard --logdir ./.

References

• Improving language understanding by generative pre-training
• Language models are unsupervised multitask learners
• Language models are few-shot learners
• minGPT

Implementation notes:

• WordPiece tokenization

Licence

MIT