microsoft / sample-app-aoai-chatGPT

Sample code for a simple web chat experience through Azure OpenAI, including Azure OpenAI On Your Data.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Chat load management by switching models

mdsa3d opened this issue · comments

Is your feature request related to this sample app, or to an Azure service, such as Azure OpenAI or Azure AI Search?

Allowing to limit using certain models for a pre-defined amount and then switch to other models for rest of the interactions.

Is your feature request related to a problem? Please describe.

I cannot switch between models during conversation.

Describe the solution you'd like

I would like to set a limit of 10 questions using GPT4, once the limit is reached then it will automatically switch to GPT3.5 for next 3 hours.

Is this feature specific to your use case or your organization, or would it apply broadly across other uses of this app?

No, everyone can use this feature, and it will allow others to manage the cost of the chatbot efficiently.

Describe alternatives you've considered

Setting the token limit at a model level.

This issue is stale because it has been open for 30 days with no activity.