Failing with Ollama
fccoelho opened this issue · comments
I am testing instructor using the basic example from the docs:
def test_get_strutured_output_ollama(self):
slm = StructuredLangModel('llama3')
response = slm.get_response('Tell me about Harry Potter', '', response_model=Character)
expected = """
{
"name": "Harry James Potter",
"age": 37,
"fact": [
"He is the chosen one.",
"He has a lightning-shaped scar on his forehead.",
"He is the son of James and Lily Potter.",
"He attended Hogwarts School of Witchcraft and Wizardry.",
"He is a skilled wizard and sorcerer.",
"He fought against Lord Voldemort and his followers.",
"He has a pet owl named Snowy."
]
}
"""
self.assertIsInstance(response, Character)
self.assertEqual('Harry Potter', response.name)But I am getting this error with llama3:
FAILED (errors=1)
/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/instructor/process_response.py:222: DeprecationWarning: FUNCTIONS is deprecated and will be removed in future versions
if mode == Mode.FUNCTIONS:
Error
Traceback (most recent call last):
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/tests/test_llms.py", line 78, in test_get_strutured_output_ollama
response = slm.get_response('Tell me about Harry Potter', '', response_model=Character)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/base_agent/llminterface.py", line 150, in get_response
response = self.llm.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/instructor/client.py", line 93, in create
return self.create_fn(
^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/instructor/patch.py", line 149, in new_create_sync
response = retry_sync(
^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/instructor/retry.py", line 160, in retry_sync
for attempt in max_retries:
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 435, in __iter__
do = self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 368, in iter
result = action(retry_state)
^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 410, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/tenacity/__init__.py", line 183, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/instructor/retry.py", line 163, in retry_sync
response = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/openai/_utils/_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 590, in create
return self._post(
^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1240, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "/home/fccoelho/Documentos/Software_projects/base-ai-agent/.venv/lib/python3.11/site-packages/openai/_base_client.py", line 1020, in _request
raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: 404 page not foundThis is how I am setting up the client, in my class:
self.llm = instructor.from_openai(
OpenAI(
base_url=os.getenv('OLLAMA_HOST', 'http://localhost:11434'),
api_key=os.getenv('OLLAMA_API_KEY', 'ollama')
),
mode=instructor.Mode.JSON
)I have the following versions of libraries installed:
openai: 1.30.5
ollama: 0.1.9
instructor: 1.3.2
openai.NotFoundError: 404 page not found
does not sound like an instructor error, you might not have started the client
I checked the ollama server is running, I ran the same query through the Ollama cli and the HTTP endpoint using curl and both worked fine.
the API endpoint v1 is undocumented the endpoints that work with curl are the /api/generate.
So I think the error is being caused by how instructor or the openai library are making the call
It may be a good Idea to add this example to your unit tests.
Update
Using the http://localhost:11434/v1 base_url and just the openai library, as shown here all works fine. So the main suspect now is the instructor.from_openai() function.
Could you please look into this? I would love to be able to use Instructor with Ollama
Seems like your test client instance used the wrong base_url: http://localhost:11434. The v1 suffix is required as indicated in the docs here.
Seems like your test client instance used the wrong base_url:
http://localhost:11434. Thev1suffix is required as indicated in the docs here.
I tried with the v1 suffix, as mentioned above, and it doesn't work either, only now instead of raising the exception, it just hangs forever.
Same error for me as well.
is ollama officially supported? if so there should be a note in the README.
we support it as much as ollama supports tool calling and
Hey. I don't know. what happened, but its working now and its great.
For me too! just ran the tests that were failing, and it is working as advertised.
Great. Please close the issue.