What about stambecco for Llama 2?

[Andrea Serrano]

I was wondering if there is a need to adapt stambecco to Llama 2. At this time if I try to generate some text using "mchl-labs/stambecco-13b-plus" on top of "meta-llama/Llama-2-13b-hf" I get the following size mismatch error
RuntimeError: mat1 and mat2 shapes cannot be multiplied (5932x5120 and 1x13107200)
which should make sense.

Should I train a similar adapter from scratch or is there a way to make stambecco compatible to Llama 2?

Thank you in advance for sharing your opinion with me,

Andrea

[Michael]

Hi Andrea,

I don't know if there is a way to make stambecco compatible with Llama 2, but I'm a bit skeptical.

Even if possible, I believe that trying to use these weights with Llama 2 won't work well due to some differences between Llama and Llama 2, especially if you are interested in making the most out of its chat capabilities. Llama 2 uses a different prompt format using [INST] [/INST] as a marker of the user input. Therefore, using weights fine-tuned using a different prompt format would result in degraded performance.

My suggestion is to fine-tune Llama 2 from zero using the stambecco italian dataset and preparing the training prompt to make it compatible with Llama 2. If you adapt the stambecco_train script for Llama 2 (take a look at SFT - Supervised Fine-Tuning), I would be happy if you open a pull request. You can then take advantage of the stambecco_demo and stambecco_chat to test out how your model behaves.

I hope it was helpful.

Michael