GPU requirements
ejarkm opened this issue · comments
Hi, first of all thanks for your work. I will definitely give it a try.
I was wondering if you could share some information about the training time and which GPUs you needed for train the model and if you have some recommendations about the size of the datasets you used.
Thanks a lot once again and merry Christmas!
This model hasn't been trained by many, since PaLM is exclusive to Google and this repo is open-source alternative.
The model size is configurable through parameters, though smaller models train faster, bigger models perform better and for PaLM it has advantage on model scaling, meaning improvements do not get stuck on bigger model size.
For pretraining datasets you can check Eleuther.ai. For RLHF fine-tuning datasets, you can check hh-rlhf from Anthropic and following-instructions-human-feedback from OpenAI.
Thanks a lot for your reply, this is really useful information :)