EleutherAI / gpt-neox

An implementation of model parallel autoregressive transformers on GPUs, based on the Megatron and DeepSpeed libraries

https://www.eleuther.ai/

Support for Mosaic Models

rajveer43 opened this issue a year ago · comments

Rockerz commented a year ago

Support for Mosaic Models

Stella Biderman commented a year ago

Mosaic models use our tokenizer, so the main to-do items are:

Are there any configs or features they use that we don't yet support?
Converting the checkpoints to the GPT-NeoX format