Add support for remote models (OpenAI, ...)
cztomsik opened this issue · comments
- Add OPEN_AI_KEY, etc. in Settings
- Update <ModelSelect, to check if this is filled and to include OpenAI models
- If any OpenAI model is selected make it visually distinctive, that you are using remote model
- Hide "partial" completion in <EditMessage if remote model is selected
- call remote endpoints
- these are different to what we are doing right now, so maybe we can first add
/api/chat/completionsendpoint which will just wrap what we do in client-side and then if we are using remote endpoint, we can just proxy
- these are different to what we are doing right now, so maybe we can first add
Would be great if Ava supported remote openai api. This would allow us to reuse the server and avoid loading the model multiple times if we are using it in a different app.
- Ollama recently added openai compatibility layer. https://ollama.com/blog/openai-compatibility
- https://edgen.co is another one that adds openai compatibility layer.
Would be really great if Ava supported using an already running Ollama instance via its API!
Would be really great if Ava supported using an already running Ollama instance via its API!
Yes, this is in the works, but not finished yet.
Just a small update, the UI part has been rewritten and we now have /api/chat/completions endpoint which is mostly compatible with openai, so hopefully, we are really close to closing this.
What's missing:
- add a new field to the Settings page for OpenAI API key
- if this is filled in, we should offer gpt models in the
<ModelSelect(it's not yet clear which ones, and how to configure that) - decide if the real api request is going to happen from the browser (simple) or from our
/apiendpoint (more work), because the first option would make the api key visible in the web browser devtools panel - disable some features if openai model is selected
Not planned anymore, but PRs are welcome.