Read Time Out after Batch Testing
deepakas opened this issue · comments
Bug Details
Describe the bug
We are using the backend API to generate response for a set of test questions. When we run for high batches the application fails.
We ran the api for 100 questions with batch size of 10 and time delay of 20 seconds. It worked 2 times. Third time application crashed. API requests to batch endpoint is giving an empty response. Web app returns this error -
Error: HTTPSConnectionPool(host='infoasst-enrichmentweb-clc9v.azurewebsites.net', port=443): Read timed out. (read timeout=60)
Steps To Reproduce
- Run Batch Testing on backend with 100 requests with a batch of 10 requests at a time.
- After multiple attempts the application crashes.
- Backend responds with empty response
What is the expected behavior? Return json response
Screenshots
ErrorsFromAppServiceLogs.txt
If applicable, add screenshots to help explain your problem.
Information Assistant details
Please provide the following details. You can simply include a screenshot of your Info panel as well.
GitHub branch: main
Version or Latest commit:
commit f4b3f3a (HEAD -> main, origin/main, origin/HEAD)
Merge: ce46332 a575426
Author: dayland 48474707+dayland@users.noreply.github.com
What region is your Azure Open AI Service in? US eastWhat ChatGPT model are you using? gpt-4
model name: gpt-4
model version: 0125-preview
What embeddings model are you using? [text-embedding-ada-002]
Additional context
Errors from App Service Logs :
If the bug is confirmed, would you be willing to submit a PR?
- Yes
Hi deepakas. How are you running batch of questions against IA? Are they not dependent on each other?( Conversational style) If you have set of questions and pairs for 100 questions would you please run this through promptflow as flow automates this for testing.
Thanks @ArpitaisAn0maly. I am making an api call to the backend chat api to get bot response and submitting batches of 10 and having delay of 20 seconds with retries with doubling the delay time. Resolved the issue by scaling the apps and restarting. Thanks George for your help and reference documentation. https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md
Resolved the issue with scaling and restarting the apps - https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md