WizardCoder hallucinations or bug in inference params?

[Extremys]

WizardCoder is really subject to expose roles from the prompt template and continue the discussion like this:

Do you have an idea what could cause this undesirable behavior? How to avoid that?
I am using the fastchat repo for inference reusing alpaca prompt template, do you see any bug?
https://github.com/lm-sys/FastChat/blob/main/fastchat/conversation.py#L378C26-L378C26

Thanks.

[ChiYeung Law]

Your template has a problem. The stop token of WizardCoder <|endoftext|> is different from Alpaca </s>.

[Extremys]

thanks for your feedback Chi, I think there is an other because switching from to <|endoftext|> do not resolve the tricky behavior :) any idea?

[ChiYeung Law]

I try to say Hi!. It seems to work well.

[ChiYeung Law]

My suggestion is to use the raw template rather than the template from FastChat.
Raw template:

Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.

### Instruction:
{instruction}

### Response:

[Extremys]

Hello Chi, I found the root cause thanks for your feedback it was coming from the worker implementation :)