Use Ollama models instead of openai
MikeyBeez opened this issue · comments
Describe the feature
I want to use a local model like llama2. The code is very simple.
from langchain.llms import Ollama
ollama = Ollama(base_url='http://localhost:11434', model="llama2")
response = ollama(prompt)
I don't know what to do to add this however. I assume connection to an llm is made in one function, but I don't know which.
Describe the solution you'd like
Can you add a section to your documentation describing how to do this?
Thanks for writing this great software!
Hello @MikeyBeez , I assume you are executing it through 'make install'. In this scenario, llm-server operates within a Docker container, while ollama runs on localhost. To adapt to this setup, set your base URL to host.docker.internal:11434.
@MikeyBeez additionally to answer the following
"I don't know what to do to add this however. I assume connection to an llm is made in one function, but I don't know which."
llm-server/utils/get_chat_model.py - this is where we initialize the chat model
llm-server/shared/utils/opencopilot_utils/get_embeddings.py - to initialize embedding model
I assume I need to edit docker-compose.arm.yaml to add the base url in the llm-server section. Would you be so kind as to give me the edit I need. I see that openchat is already and option in get_chat_model.py. Do I edit the model_name to = "openchat" Similarly, do I edit embedding_provider to equal openchat?
Thank you so much for your kind attention. <3
Or do I just make an environment variable called base_url?
Hey @MikeyBeez ,
If you don't mind, we can give you a 1-1 support on discord. Maybe even hop on a video call to figure this out together. https://discord.gg/8FUnVjGEd5
To answer the question:
The base url is configurable through an environment variable
- CHAT_MODEL=model_name
- LOCAL_IP=host.docker.internal # to setup the base url
- EMBEDDING_PROVIDER=embedding_provider_name
Let me know if you can join discord, or else I can setup a google meet / zoom