Issues when finetuning pretrained lora checkpoint for downstream tasks.

[Prateek Yadav]

Hi Lucas,

I was trying to run the MHR codebase to finetune on a downstream task after loading the pretrained lora checkpoint provided here. I am not able to load the pretrained lora checkpoints into the model and I getting some shape mismatch errors. I am pretty sure I am messing up something with the config files and hyperparameters setting. Would it be possible for you to provide a command that can be used to FT lora checkpoint on any downstream task say RTE?

I was able to load the poly checkpoint and finetune that further using the provided scripts here. It would be great if you can add scripts to finetune lora, ia3, and mhr pretrained checkpoints.

Thanks,
Prateek

[Lucas Caccia]

Hi Prateek,

Sorry for the late reply; did you manage to solve this issue ?