Fine-tuning recommendations
RainIwakura opened this issue · comments
Hello EleutherAI team! Congrats on your ICML acceptance for this paper. I'm a PhD student intending to study emergent properties of GPT-type of decoder models using Pythia for my final paper, and was wondering if you have any insight into what sort of learning rate, data format, or optimizer you'd recommend on using to achieve "optimal" results for fine-tuning these models on custom small datasets (rules conditioned on size perhaps), at least in your experience.
When I say small I mean at most 100k training points (pieces of text) of around 60 tokens each in terms of length if that helps. Anyways, I'd welcome any information! Apologies if I missed anything obvious within the repo that'd point me to this info. It seems like the majority of the information is on pretraining and I do not intend to do that.
Hi! We haven't finetuned these models much at all yet ourselves, so not confident on hyperparameter choice for best results. I'd recommend trying out a batch size of 256 or 128 points, and starting learning rate at (0.1x the max LR used for pretraining), decaying on a cosine schedule to 0.1x this starting LR plus linear LR warmup, as a starting point! But not sure of the optimality of this or if it will work on your dataset.
Thank you! Some starting pointers is all I needed.