yhyu13 / GPT-RMT

An experiment to test Recurrent Memory Transformers in GPT

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

GPT-RMT - An experiment to test Reccurent Memory Transformers in GPT models

  • More info coming soon

Updates

  • May 5th, 2023: Trained a demo model with mixed results. Unfortunately, adding an RMT makes training incredibly slow, which slows down hyper param tuning. Will be hopefully posting results within the next two weeks

About

An experiment to test Recurrent Memory Transformers in GPT