HTTP Server closed connections keep being in CLOSED_WAIT state
jgaalen opened this issue · comments
Expected behavior
If the server closes an HTTP connection, jmeter should handle the closed connection immediately and not keeping it in CLOSE_WAIT state until the next request, which could also lead to "Non HTTP response" errors
Actual behavior
Currently, when JMeter does an HTTP request and is in keep-alive state and server waits for the next request. If the server closes the connection after X-seconds and sends a FIN/ACK packet, JMeter does nothing. If the next request is done within 5 seconds after the FIN/ACK packet, it sets a "Non HTTP response message: : failed to respond" error immediately, because the server is already in CLOSED state and can't pick up new packets. This results in this error.
If it tries to send a request later than 5 seconds, somehow it gracefully sends a FIN/ACK back and creates a new connection and does the request.
Steps to reproduce the problem
Set keep-alive timeout to 2 seconds on the server, so it shuts down the connection after 2 seconds.
Then create a jmeter script to send request 1, then wait 3 seconds, then send request 2. It will end up with the Non HTTP response error, because it tries to send the request on a closed socket.
Now wait 6 seconds rather than 3, and somehow it does work without an error.
JMeter Version
5.6.3
Java Version
17
OS Version
Mac + Ubuntu
I've done some more testing on this. It doesn't turn into a non-HTTP error against every server so I have no idea why that happens with one server I'm using, but the problem remains. On many servers somehow, after the FIN_ACK is sent and jmeter leaves it in CLOSE_WAIT state, when JMeter wants to do a new request, it somehow first sends the FIN_ACK back which cleans the connection and then does the request. Against some servers it doesn't send the FIN_ACK when it wants to do a new request, but just sends it over the connection and the server sends a RST because the connection is either cleaned up or it's in CLOSE_WAIT state.
It is a serious issue, because once the server tries to disconnect the connection (after its keep-alive timeout), the connection remains in CLOSE_WAIT state on both the webserver and jmeter client and still taking/tracking a socket which should not be there. Browsers clean up the socket immediately, so it's not there anymore on both the client and the server which is the behaviour we need to see in JMeter
I've made a setup with basic nginx, keep-alive timeout changed to 2 seconds. Then a script that does a request (with keep-alive), then 4 seconds later, an new requests. The 2nd requests fails due to this RST and connection in CLOSE_WAIT state.
It only happens with the HttpClient4 implementation. The Java implementation has no issues
So I've found the root cause of the issue and it's due to this config httpclient4.validate_after_inactivity.
This defaults to 4900 (ms), meaning that if a connection trying to be disconnected by the server within 4900ms, and a new request is done between the requested disconnect and 4900ms, it just sends the request packet over the connection in CLOSE_WAIT state, resulting in a connection reset error.
I think this is a bad implementation in the HttpClient4 library. It should simply remove a connection once the server requests it.
A work-around is to set httpclient4.validate_after_inactivity low (900). This only resolves the unexpected "Non HTTP response" errors, but doesn't resolve the high CLOSE_WAIT state taking unrealistic higher connections on jmeter clients and left FIN_WAIT_2 state on webservers
I experienced the same issue - thanks for posting here!
Should a ticket be added for this? Decreasing the validation period helps but as you mentioned doesn't resolve the high CLOSE_WAIT states etc.
I experienced the same issue - thanks for posting here! Should a ticket be added for this? Decreasing the validation period helps but as you mentioned doesn't resolve the high CLOSE_WAIT states etc.
Well I created a ticket with this right? But indeed, it should actually be resolved and behave like a browser, or any http client. So it should respond to the FIN packet and close the socket on the jmeter side and send a FIN back, so the server can remove the half closed state as well
Ah you are right - I thought it is still the bugzilla tracker... But since 2022 it was migrated here. So everything is fine :-)