A attempt to write finetuning script for codegen

Question

A attempt to write finetuning script for codegen

vishalsingha opened this issue 8 months ago · comments

Hii,
I am interested in writing finetuning script for this model.

Can anyone tell while training the model in what format can i provide output to the model.

As the data is of form
{code:"def hello(name): return f"Hello {name}", nl : "This function takes name as input and a massage saying hello to the person in format hello name."}

We can tokenize input (either code or text ) by the making tokenizer and passing text into it.

But what format should I use for training and how to compare loss of original output and predicted output.

Thanks

VISHAL SINGH · Answer 1 · Mon Jun 06 2022 12:53:30 GMT+0800 (China Standard Time)

Hello,
Can anyone give me the small sample (only 4 to 5 examples ) of BIGPYTHON dataset used for training the Codegen-nl mono just to get idea of training set.

nashid · Answer 2 · Thu Jun 09 2022 13:53:19 GMT+0800 (China Standard Time)

@vishalsingha wondering whether you have made any progress in this regard?

nashid · Answer 3 · Wed Jul 20 2022 11:08:39 GMT+0800 (China Standard Time)

@vishalsingha do you plan to share it?

VISHAL SINGH · Answer 4 · Wed Jul 20 2022 16:14:14 GMT+0800 (China Standard Time)

Sorry it was done as part of internship so according to company policies I can't share it.

nashid · Answer 5 · Fri Jul 22 2022 08:45:12 GMT+0800 (China Standard Time)

@vishalsingha got it. Would be awesome if you share any insight how you approached developing the fine tuning script. Also any reference to open source resources would be much appreciated.