There are two app containers which have a FastAPI server running and an nginx container to load balance the requests.
- The very initial version which could properly load balance was keeping around 400-500 TCP connections open but but luckily the open TCP connections inside the container running uvicorn were very low since nginx forwarded requests with header "Connection: close" which would eliminate the connection with uvicorn.
- The second version with additional configs around TCP connection resuse and keep-alives could manage to keep the open TCP connections at both the containers (nginx, uvicorn) in under 10.