Toy-GPT is a decoder-only transformer built from scratch using NumPy and PyTorch, trained to generate natural language similar to input.txt
- Python3
- PyTorch:
conda install pytorch torchvision -c pytorch
- Training takes ~1 hr on Apple's M1 Pro Chip
- Language generation quality is limited by compute
Trained on Shakespearean text:
SICINIUS:
Is it strange?
Herald:
He's deceited, and children from his new spid
Then whomen he dares to him: were he worse.
BRUTUS:
You have pirtly not him.
MENENIUS:
What's the prisoner have not a silfa?
MONTAGUE:
O, and both shame, Menenius. Stanless, Thou art purpose;
And said thou pen for thy melting there,--
BENVOLIO:
Two sir, the earth proofs rids too come hither;
I thank you out, as thought sook for Ireland,
FRIAR LAURENCE:
His son, do your morself, that leaven your honours
Sufferable in more and suffer five.
A horse! High-graced York rights. And bother Montague