Can u opensource the training script which can reproduce llama3 result?
lucasjinreal opened this issue · comments
Looks like the internal codebase didn't actually opensource.
What you mention has been supported 21483e8
We are super busy these days but the training strategy is scheduled to be released. Stay tuned.
Does the vit opened both in pretrain and sft?
The strategy only differs in the visual instruction tuning stage. And the vision tower was frozen under pre-training stage.
Does the llama3 instruct chat format support?
Bunny-Llama-3-8B-V was trained under "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\n{prompt} ASSISTANT:" template.
You can try it to see whether it works well under the origin template.
Oh, that's violated the llama8b-instruct's template, doesn't it will harm original language ability?
We use the same template for all bunny models for consistency and convenience. The performance is also acceptable.
Close the issue for now if there's no further discussions. Feel free to reopen it if there's any other questions.
@lucasjinreal We released all of the training strategy and data of latest Bunny. Check more details about Bunny in Technical Report, Data and Training Tutorial!