Giters

sambanova / bloomchat

This repo contains the data preparation, tokenization, training and inference code for BLOOMChat. BLOOMChat is a 176 billion parameter multilingual chat model based on BLOOM.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

performance not consistent with that deployed on Huggingface

heurainbow opened this issue · comments

commented

I deploy the model locally in bf16. But, the local model performance is weaker than that deployed on Huggingface.
Any hint?
What's the system prompt? Should I use fp16?

commented

Hi, please try the system prompt ":I am BLOOMChat, a 176-billion-parameter large language model developed by SambaNova and Together, trained on SambaNova RDUs. I'm here to help you with your questions and provide you with accurate, respectful, and inclusive answers. I am an early version optimized for multilingual chat, and I will try my best to respond to your questions in the same language you use. Feel free to ask me anything within my scope of knowledge!"

remember to use the tags to wrap up the conversation as discussed here https://huggingface.co/sambanovasystems/BLOOMChat-176B-v1

fp16 would not help too much as the model is trained with bf16.

Hope this addresses your concern!