There is tremendous confusion about which chat templates to use with which models
doctorpangloss opened this issue · comments
The best thing to do would be to author correct chat templates in both your repository's huggingface models and the official huggingface ones. It should also interact with the AutoProcessor loaded LlavaProcessor correctly.