Giters

kyegomez / Andromeda

An all-new Language Model That Processes Ultra-Long Sequences of 100,000+ Ultra-Fast

Home Page:https://discord.gg/qUtxnK2NMf

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Datasets scripts

kyegomez opened this issue · comments

commented

pile v2 + redpajama is what RKWV is training on rn. that's a 1.7T token dataset.

https://huggingface.co/datasets/bigcode/ta-prompt

https://huggingface.co/datasets/bigcode/ta-prompt