Giters

enricoros / big-AGI

Generative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.

Home Page:https://big-agi.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[BUG] Allow user to manually define context wnidow for unknown models

lenaxia opened this issue · comments

commented

Description

Conversation from discord:

Lenaxia — 05/09/2024 11:01 PM
Is there a way to forcefully define context windows for unknown models?

I've got BigAGI pointed to AWS Bedrock by using LiteLLM as a proxy by creating a OpenAI #2 service in BigAGI. However I defined some custom models like for Claude Sonnet and Mixtral 8x7b, but these are showing in the BigAGI models window as ?: Unknown, please let us know the ID. Assuming a 4097 context window size. Obviously Claude has much more than a 4097 context window, I just can't figure out a place to change it for unknown models.

Enrico — 05/10/2024 4:09 AM
We don't have that feature. Would be a nice addition indeed, and something not that hard to do. As the API does not communicate the name/description/window, there are 2 possibilities: either the developer hardcodes the model_id -> metadata mapping, or we give the user a UI to change the context window (and max output window) numbers.
You should create a bug about it - we have a bug about ignoring context windows and never setting them in the UI or model calls, but not a ticket to manually specify the context windows.

Bascially, in the model management window, allow the context length to be editable by the user so they could change it manually if they need to.

Device and browser

N/A

Screenshots and more

N/A

Willingness to Contribute

  • 🙋‍♂️ Yes, I would like to contribute a fix.