Clarity on training data for each of the codegen versions
sundar7D0 opened this issue · comments
In the "lessons learnt from codegen2" paper, it's discussed that data mix of pile and thestarcoder data is a better choice to undertake if enough compute is available, but it's not clear if codegen2 or codegen2.5 (base models not instruct models) were trained with natural language data like ThePile etc. Is there any small model <=7B which is trained on both ThePile and TheStarCoder data?