Model Merging
okpatil4u opened this issue · comments
Hello Eric,
This issue may not be relevant to this repo, but it seems like model merging is gathering some speed. Have you seen any examples ? Any tips on how to implement this in candle ?
Thanks !
@okpatil4u, thanks for your interest! If you mean LoRA model weight merging, candle-lora has already implemented it. Otherwise, could you please let me know what you meant by model merging?
Closing so that it does not become stale, please feel free to reopen!
Apologies Eric. I was thinking about the following repos.
https://github.com/yule-BUAA/MergeLM
https://github.com/cg123/mergekit
The idea behind is taking two different finetuned models with the same origin and merge them so their expertise could be compounded. This could be pretty useful tool if its theoretical efficiency is proven.
Just giving you a heads up.
Do you mean something like Mixtral?
@okpatil4u, I just took a look at the following link: https://github.com/cg123/mergekit. I think it would definitely be possible to implement using candle-lora's trait-based swapping mechanism!
Closed so it doesn't become stale. Please feel free to reopen if you have any ideas!