Getting the following exception 500: Expecting value: line 1 column 1443 (char 1442)

[Yves Charorn]

Hello guys,
Thank you for this awesome project. I'm getting this exception: 500: Expecting value: line 1 column 1443 (char 1442). I'm using a local Ollama (0.1.39) docker instance with the llama3 model. Here is the details log:

INFO: 172.25.0.1:52334 - "POST /chat HTTP/1.1" 200 OK
Search provider: searxng
Traceback (most recent call last):
File "/workspace/src/backend/chat.py", line 111, in stream_qa_objects
async for completion in response_gen:
File "/workspace/.venv/lib/python3.11/site-packages/llama_index/core/llms/callbacks.py", line 280, in wrapped_gen
async for x in f_return_val:
File "/workspace/.venv/lib/python3.11/site-packages/llama_index/llms/ollama/base.py", line 408, in gen
chunk = json.loads(line)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/init.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1443 (char 1442)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/workspace/src/backend/main.py", line 97, in generator
async for obj in stream_qa_objects(chat_request):
File "/workspace/src/backend/chat.py", line 140, in stream_qa_objects
raise HTTPException(status_code=500, detail=detail)
fastapi.exceptions.HTTPException: 500: Expecting value: line 1 column 144

Thanks for your help,

[ProjectMoon]

I am experiencing basically the same error, with searxng. Results stream in and then blow up with this: 500: Expecting value: line 1 column 4091 (char 4090) . Wonder if this is a problem in the ollama library, given that it's happening in site-packages.

Edit: answers sometimes complete, but not often.

[Simsk8er]

Same thing except 500: Expecting value: line 1 column 2891 (char 2890)

[Louis Radtke]

I'm sure, it isn't the perfect solution, but might be a quick fix for you guys. In my setup using the docker-compose.dev.yaml, the error appears in chat.py, ln. 111. So surrounding it with a try-catch lets me at least continue my conversation with farfalle. I just had to rebuild and -start the compose stack (docker compose -f docker-compose.dev.yaml up --build). But without digging deeper, I can't tell if I'm breaking anything else. Here's my fixed code:

        try:
            async for completion in response_gen:
                full_response += completion.delta or ""
                yield ChatResponseEvent(
                    event=StreamEvent.TEXT_CHUNK,
                    data=TextChunkStream(text=completion.delta or ""),
                )
        except Exception as ex:
            print('ERROR:\terror completing the stream')

[ProjectMoon]

While that stops the page from crashing, it suppresses the entire answer from the LLM. A better way to do it would be to put the try-catch inside the loop, but I'm not sure that's possible, because I think it is deserializing the JSON when yielding. That is, the error happens before the completion variable is even assigned to the loop iteration.