Short responses
Akiwuff opened this issue · comments
I'm getting short and truncated responses. The generated answer stops in the middle of a sentence, usually after 60 - 90 words. CPU usage drops to 0, and I can see in the log that the response in the web interface is indeed identical to the output generated. I'm using the 13B model and I'm running on windows 10. So far, I have tried to change the temperature to 0.8, and the self_repeat to 128, with no apparent effect on the behaviour.
I've got 32gb of RAM in this machine, and the memory isn't full when the generator is running.
Anyone got a hint for me how to fix this? TIA.
While I'm not able to provide an answer, ggerganov/llama.cpp#71 has done a lot thinking about this. There seem to be some upstream improvements, too. Maybe they can be applied to @ViperX7's fork, too?