Empty response when a manager is not available
TilBlechschmidt opened this issue · comments
🐛 Bug description
When the proxy selects a manager and this manager is unreachable for whatever reason it replies back to the client with an empty response.
🦶 Reproduction steps
Steps to reproduce the behavior:
- Deploy a grid to docker
- Kill the manager container
- Use
redis-clientto create a dummy manager that does not exist - Send a new-session request to the grid
🎯 Expected behaviour
The proxy server is expected to check availability of upstreams before forwarding data to them and fall back to another upstream if it isn't available. If no upstream is available a reasonable error message should be returned.
Context
Version
This bug is not version related.
Where did the problem occur?
- ☸️ Kubernetes
- 🐳 Docker
- 👨💻 Locally
Which browsers cause the bug?
This bug is not browser related
Additional context
This problem is not limited to managers and can be observed with the API and Storage components as well.
Solution sketch
There is a basic and advanced solution to this problem.
Basic solution
The simplest fix would be to wrap the error handling in the forward method of the proxy server and map the error to some reasonable message and error code.
Stretch goal
An advanced solution would be to try connecting to a different upstream if a selected one is not available. This requires modifying the RoutingInfo to return a random iterator instead of one element and the proxy job to retry with the next iterator element if an error is encountered. Note however that regular HTTP errors returned by the manager should not be filtered!