Azure bot sometimes returns Bot timeout (502, 504) error but also getting right response
dkulkarni-ms opened this issue · comments
Github issues should be used for bugs and feature requests. Use Stack Overflow for general "how-to" questions.
Version
What package version of the SDK are you using.
4.19.3 - Microsoft.Bot.Builder.*
Client side - "botframework-webchat": "^4.15.7",
"botframework-webchat-component": "^4.15.7",
Describe the bug
We have azure bot using as assistant and calling LLM APIs in the backend. On the client side we use directline from botframework to call into bot. Sometimes the GPT calls take longer time. We see that activities call timesout from network tab sometimes and returns 504 error. However, we do get the right response later from the backend. In UI we dont see any error message as timeout and we see next correct response from the backend displaying in UI. Its good that UI doesnt show any error and waits for bot to finish. We want to understand though is there any internal logic to ignore 504/502 errors and give more time for Bot to send the right response?
To Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior
We should not see timeout for activity call. Rather should see that all calls are returning 200.
Screenshots
Additional context
Add any other context about the problem here.
Tracking Status
Dotnet SDK TODO
- PR
- Merged
Javascript SDK TODO
- PR
- Merged
Python SDK TODO
- PR
- Merged
Java SDK TODO
- PR
- Merged
Samples TODO
- PR
- Merged
Docs TODO
- PR
- Merged
Tools TODO
- PR
- Merged
Hi @dkulkarni-ms,
Direct Line enforces a service-side timeout of 15 seconds that cannot be modified. In order to eliminate these bot timeout errors, you will need to either perform LLM calls outside the bot or use asynchronous Tasks to perform the long running LLM calls on a background thread. Please take a look at this sample for a starting point on implementing this with background threads.