[BUG] It seems Vercel deployments do not return stream for Ollama

[Daniel Farina]

App works fine locally but on Vercel the chat only returns the first string and freezes. Is there a different configuration or something that am I missing for vercel?

Thanks!

[Jorge Menjivar]

It should work fine without any extra config. What model are you using?

[Daniel Farina]

After deeper debugging the issue lies on the ollama stream.js where I was getting this error on the server side:

Error parsing JSON SyntaxError: Unterminated string in JSON at position 28 (line 1 column 29)
    at (utils/server/ai_vendors/ollama/stream.ts:114:36)

const splits = decodedText.split('}\n{');
for (let i = 0; i < splits.length; i++) {
  let split = splits[i];
  if (i !== 0) {
    split = '{' + split;
  }
  if (i !== splits.length - 1) {
    split = split + '}';
  }
  const parsedData = JSON.parse(split); // This line could throw the error if 'split' is not valid JSON.
  // ... rest of the loop
}

In the above code, the string is being split on every '}\n{'. This implies an assumption that the chunks are always going to be nicely divided along JSON object boundaries, which may not be true. If a chunk ends (or starts) in the middle of a JSON object, it will not be a valid JSON after the split and concatenation.

To fix this issue, you would need to ensure that the chunks are being reassembled correctly into valid JSON before attempting to parse them. One way to do this could be to buffer the chunks until you have a complete JSON structure before parsing. If the API is streaming JSON objects, you might need to implement logic to handle partial objects.

Submitted a PR. Ollama working locally and on Vercel now.

#81

[Jorge Menjivar]

Just merged it, thank you! Would that include the final fix?

[Daniel Farina]

that's it, thanks!