some chunks received are not full json
JerrySievert opened this issue · comments
when ollama is running remotely, some chunks received are not actually fully formed json when they returned. we should not be attempting to decode them until they are complete, or perhaps use a steaming json parser.
either way, it ends up making dumbar think that the connection is disconnected because of the error thrown.
fix it!
@JerrySievert love this project! Thanks for building it!
Sorry Ollama is returning incomplete json objects. Wondering if you'd be able to help track down why this might be happening: what does the data being returned look like? and do you have an easy way to reproduce it that you can share with me / the Ollama maintainers? Will try to reproduce it on our end as well and get this fixed 💪
I don't think it's ollamas fault, actually, but if it is, I will log it and maybe we can figure it out together!
my hypothesis is that in the case of a remote machine hosting ollama, connected to a gigabit network that is connected to a wifi router, that is sending the data to another mesh network AP, and then to my laptop, the packets are getting separated just enough that it thinks the read is complete in the client; and, since the reader (https://github.com/JerrySievert/Dumbar/blob/main/app.js#L19-L23) doesn't know what a json packet looks like, it's terminating the read before the whole json packet has been drained.
when I looked at the web inspector, the json bits appeared to be ok on first glance.
regardless, this is mostly just a note to myself to fix it when I get time.
thanks for your awesome work on ollama, btw! it definitely makes for more awesome fun! I have a feature request, but I'll get the feature figured out and open an issue (probably not a PR, because it's non-trivial and you'll want to make real design decisions around it). keep up the great work, and I look forward to what comes next!
I haven't checked packet length, or what the MTU is set to through every hop. it works amazingly locally, so my theory is likely close to reality.
as expected, the client was not receiving complete json results per what it considered a response. switched to a streaming json parser, and gosh if things don't just work how they're supposed to.