Something about User: not stopping the conversation.
leucome opened this issue · comments
Ok I used Amica with Ollama on CPU with model like mistral dolphin-mistral and Luna llama2. Everything looked fine.
Then I installed Kobold with rocBLAS. Then Amica started to respond in my place like, User bla bla bla.
I am 99% sure that User: was hard coded to stop the conversation and wait. The only explanation I can think of is that the delays for stopping the conversation is too long Is there no stopper maybe? With rocBLAS I get over 60T/s on q4 model. Then it seem to be enough to generate a lot of sentence before any stomping can happens.
I tried the demo and noticed that sometime the demo is able to say User: Yes in my place for yes no question then stop just after that. It seem to support my hypothesis that it do not stop, or stop fast enough for short sentence.
So it it plausible? If so how can we make the stopping function that kick in quicker when the AI try to respondent for the user?
Also related to the LLM trying to take user places. Is there a prompt that is known to reduce the chance that it happen.
Even with llama.cpp sometime the LLM slip and try to impersonate the "User" by not using "User:"
Also I tried to find where the "User:" stop word it is in the code. but searching the word user give quite a lot of result. I though I found it in /src/features/chat/llamaCppChat.ts . But when i added "User" there alongside "User:" it did not make it stop at User.
I tried this
headers: headers, method: "POST", body: JSON.stringify({ stream: true, n_predict: 400, temperature: 0.85, cache_prompt: true, stop: [ "</s>",${config("name")}:, "User:", "User"
Arbius has a $100 AIUS bounty for this issue!
Brief: Figure out what is causing the bug and fix it.
Please read carefully:
To begin work on a bounty, reply by saying “I claim this bounty” - you will have 48 hours to submit your PR before someone else may attempt to claim this bounty.
To complete the bounty, within 48 hours of claiming, reply with a link to your PR referencing this issue and an Ethereum address. You must comply with reviewers comments and have the PR merged to receive the bounty reward. Please be sure to focus on quality submissions to minimize the amount of time reviewers must take.
Thank you. Your PR has been merged. Please send ETH address for AIUS.
Thank you. Your PR has been merged. Please send ETH address for AIUS.
0xD44248b6Ab4B6580cd982A80D4B26dC7058264f2