Request: Add OPENAI_API_URL
azdolinski opened this issue · comments
-
Please consider adding the environment variable OPENAI_API_URL. This addition will facilitate communication with LiteLLM, which adheres to the OpenAI API protocol and acts as a local proxy.
Through this configuration, you'll also gain the capability to connect to Ollama, enabling local LLM interactions. -
LiteLLM can be deploy as container... and support API method /models [/v1/models]
so you can also add read list of models...
curl -X 'GET' \
'http://localhost:4000/v1/models' \
-H 'accept: application/json' \
-H 'Authorization: Bearer sk-XXXXXXXXXXX'Example reponse:
{
"data": [
{
"id": "together_ai-CodeLlama-34b-Instruct",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-4-turbo",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "anthropic-claude-3-haiku-20240307",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-4",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "google-gemini-1.5-pro-preview-0409",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "perplexity-mistral-7b-instruct",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "huggingface-zephyr-beta",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "together_ai-CodeLlama-34b-Python-completion",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-whisper",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-3.5-turbo-16k",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "ollama-mistral-7b",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "google-gemini-pro",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "groq-llama3-70b-8192",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "anthropic-claude-3-opus-20240229",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-4-vision-preview",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-4-32k",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "groq-llama3-8b-8192",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "huggingface-Xwin-Math-70B-V1.0",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-4o",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "groq-mixtral-8x7b-32768",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "google-gemini-1.5-flash-preview-0514",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "openai-gpt-3.5-turbo",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "anthropic-claude-3-sonnet-20240229",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "ollama-mxbai-embed-large",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "groq-gemma-7b-it-8192",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
},
{
"id": "perplexity-mixtral-8x22b",
"object": "model",
"created": 1677610602,
"owned_by": "openai"
}
],
"object": "list"
}
Benefits:
- Facilitates seamless interaction with local instances of LiteLLM.
- Enables the use of local LLMs through Ollama.
- Increases flexibility in choosing between cloud-based and local AI resources.
Thank you for considering this enhancement.
hey! I just added support for local models through ollama. This doesn't support all of the models you listed, but let me know if this is helpful
@rashadphz
My request is not directly releted to local ollama..
LiteLLM hosted locally in docker, allow you connect to multiple vendors (including ollama). It open for you support more then 100+ LLMs, load balance, cost tracking etc.
https://docs.litellm.ai/docs/providers
[[farfalle]] ---(OpenAI API)--->[[LiteLLM (docker)]] ---(vendor API)----> [[OpenAI/Gemini/Groq/(ollama) etc..]]
For me - it allow in easy way manage (route logic) connections to multiple ai-clouds/vendors (and keep all api-key in one place).
Got it, just added this to the roadmap.
@rashadphz Great Job!
Would you like to add deepseek to the code?
I did not find the code how to change api.openai.com to api.deepseek.com T T
If you would like to add this one to the repo, that would be great, or would you like to change the code so that we could change the url of openai locally?
The V2 of deepseek is super cool (In my Chinese literature and logic test, deepseek performs better than gpt-4 and llama3 70b), it's doc is like this:
The DeepSeek API utilizes an API format compatible with OpenAI. By modifying the configuration, you can use the OpenAI SDK to access the DeepSeek API, or employ software that is compatible with the OpenAI API.
Parameters Values
base_url * https://api.deepseek.com/
model deepseek-chat
For compatibility with OpenAI, you may also set the base_url to https://api.deepseek.com/v1 to use.
@rashadphz Great Job! Would you like to add deepseek to the code? I did not find the code how to change api.openai.com to api.deepseek.com T T
If you would like to add this one to the repo, that would be great, or would you like to change the code so that we could change the url of openai locally?
The V2 of deepseek is super cool (In my Chinese literature and logic test, deepseek performs better than gpt-4 and llama3 70b), it's doc is like this:
The DeepSeek API utilizes an API format compatible with OpenAI. By modifying the configuration, you can use the OpenAI SDK to access the DeepSeek API, or employ software that is compatible with the OpenAI API.
Parameters Values
base_url * https://api.deepseek.com/ model deepseek-chat
For compatibility with OpenAI, you may also set the base_url to https://api.deepseek.com/v1 to use.
By the way, the hf page of deepseek v2 is here:
@getofferhelp @azdolinski I found that you can refer to this link to modify line 50 in chat.py within the backend, something like this: return llm = OpenAILike(model="my model", api_base="https://hostname.com/v1", api_key="fake"). Additionally, you need to add OpenAILike to the dependencies. It should be possible to modify the api_base. I plan to try it out first to see if it works.
Link: https://docs.llamaindex.ai/en/latest/api_reference/llms/openai_like/
@getofferhelp @azdolinski I found that you can refer to this link to modify line 50 in chat.py within the backend, something like this: return llm = OpenAILike(model="my model", api_base="https://hostname.com/v1", api_key="fake"). Additionally, you need to add OpenAILike to the dependencies. It should be possible to modify the api_base. I plan to try it out first to see if it works.
Link: https://docs.llamaindex.ai/en/latest/api_reference/llms/openai_like/
Great! Thank you so much! I would try it too.
@getofferhelp @azdolinski I found that you can refer to this link to modify line 50 in chat.py within the backend, something like this: return llm = OpenAILike(model="my model", api_base="https://hostname.com/v1", api_key="fake"). Additionally, you need to add OpenAILike to the dependencies. It should be possible to modify the api_base. I plan to try it out first to see if it works.
Link: https://docs.llamaindex.ai/en/latest/api_reference/llms/openai_like/
I was able to get it working. You don't need OpenAILike, but there are 2 things you need to change.
Line 50 in chat.py needs to be return OpenAI(api_base="https://hostname.com/v1", model=model_mappings[model])
and
Line 24 in related_queries.py needs to be openai.AsyncOpenAI( base_url="https://hostname.com/v1", api_key="fake", ),
From here, you are able to change the name of the model in constants.py, and it should work.
I am having the same problem and changed the lines you suggested and now I get: 500: Error code: 401 - {'error': {'message': 'Incorrect API key provided: fake. You can find your API key at https://platform.openai.com/account/api-keys.','type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}
Which is wierd, as my AI server doesn't need an API key.