Many sockets in FinWait2 and CloseWait state
datwelk opened this issue · comments
When I instantiate many connections (~30 per second) to the same server from iOS, at some point I quickly receive a connectionFromPoolTimeout for new connections.
At that point the network profiler shows many sockets in FinWait2 or CloseWait state. When connectivity recovers, the FinWait2 and CloseWait sockets are gone.
Is this an issue on the server side, where the server does not gracefully close connections, or could this be a client side issue?
I am using the NIOTSEventLoopGroup.
HTTPClientError.getConnectionFromPoolTimeout indicates that a request timed out while at least one active connection is alive. Active in this context means that it was at some point established and has not yet been closed.
I'm sadly not familiar with the socket connection state. It sounds like this type of question is a more general problem of TCP connections in general on iOS and might be better asked on the Network.framework specific Apple forum or via a Technical Support Request.
The open sockets seem to be unrelated.
I do keep running into the HTTPClientError.getConnectionFromPoolTimeout though. I have a minimum reproducible case. The target server doesn't matter.
If you run the sample app, at some point the console will be full of "Failure: HTTPClientError.getConnectionFromPoolTimeout".
This is expected because you do not have any backpressure. Therefore more request per second get spawned then executed and the queue gets longer and longer. Eventually the requests will timeout and you get HTTPClientError.getConnectionFromPoolTimeout as an error.
You can either increase the timeout or limit the number of requests you are doing in parallel.
How could I limit the number of requests in parallel?
I use a PriorityQueue to manage outgoing requests before passing them on to the AsyncHTTPClient. When I reduce its capacity from 20 to 3, the errors seem to go away. What's the recommended number of parallel requests per host?
This depends if HTTP/1 or HTTP/2 is used. If HTTP/1 is used the maxHTTP1Connection config says how many requests can run in parallel. If HTTP/2 is used, the server dictates the max parallel requests (called streams in the spec) which can change over time.
Something like TokenBucket from SwiftPM can be used to limit the maximum requests.
I think at the end we should add some API to allow to easily keep the connection busy without filling up the queue. However, this is not possible today.
Sounds good. I will implement a max req per host limit on my PriorityQueue wrapper. Thank you for your assistance! @dnadoba