Connection is not shut down cleanly when using HTTPS

Question

Connection is not shut down cleanly when using HTTPS

MrMage opened this issue 6 years ago · comments

When connecting via HTTPS, I see the following errors in the console:

[HTTP] HTTPClient: uncleanShutdown [HTTPClient.swift:30]

I think the solution to this would be to have e.g. HTTPClient.close() call OpenSSLHandler.close(ctx: ChannelHandlerContext, mode: CloseMode, promise: EventLoopPromise<Void>?) in case of HTTPS connections.

Tanner · Answer 1 · Thu May 03 2018 01:24:12 GMT+0800 (China Standard Time)

@MrMage I'm not able to reproduce this error in tests. Could you supply some code that reliably creates it?

Daniel Gräfe · Answer 2 · Thu May 03 2018 01:35:17 GMT+0800 (China Standard Time)

I'm on the go right now. For me, simply connecting to an HTTPS server and sending one request causes the error once the client is closed. Did you try with HTTPS and ensure that the client is closed/deallocated?

Daniel Gräfe · Answer 3 · Sat May 05 2018 20:12:18 GMT+0800 (China Standard Time)

Here is some sample code that fails 100% for me (might be specific to some hosts):

do {
	let eventLoopGroup = MultiThreadedEventLoopGroup(numThreads: 1)
	let eventLoop = eventLoopGroup.next()
	let httpClient = try HTTPClient.connect(scheme: .https, hostname: "accounts.google.com", on: eventLoop).wait()
	try httpClient.send(.init(method: .POST, url: "https://accounts.google.com/o/oauth2/token",
							  headers: .init([("content-type", "application/x-www-form-urlencoded")]),
							  body: Data())).wait()
	try httpClient.close().wait()
} catch {
	dump(error)
}

This fails with the following messages:

- NIOOpenSSL.OpenSSLError.uncleanShutdown
[ERROR] [HTTP] [HTTPClient] uncleanShutdown [HTTPClient.swift:32]

Even if it's in part due to the host (although I wouldn't expect Google's servers to do something non-standard there), I think there should be an option to initiate a clean shutdown from Vapor, as mentioned above.

Daniel Gräfe · Answer 4 · Wed May 23 2018 20:02:43 GMT+0800 (China Standard Time)

@tanner0101 were you able to reproduce this with my instructions? And would the fix outlined in the first post be appropriate?

Tanner · Answer 5 · Wed May 30 2018 09:34:47 GMT+0800 (China Standard Time)

I haven't been able to re-create yet. A failing test case would help a ton here.

Daniel Gräfe · Answer 6 · Wed May 30 2018 18:51:33 GMT+0800 (China Standard Time)

@tanner0101 I have created https://github.com/vapor/http/compare/httpclient-openssl-error?expand=1, which gives the following log output for me:

Test Suite 'Selected tests' started at 2018-05-30 12:47:48.063
Test Suite 'HTTPTests.xctest' started at 2018-05-30 12:47:48.064
Test Suite 'HTTPClientTests' started at 2018-05-30 12:47:48.064
Test Case '-[HTTPTests.HTTPClientTests testAmazonWithTLS]' started.
[ERROR] [HTTP] [HTTPClient] uncleanShutdown [HTTPClient.swift:32]
<unknown>:0: error: -[HTTPTests.HTTPClientTests testAmazonWithTLS] : failed: caught error: The operation couldn’t be completed. (NIOOpenSSL.OpenSSLError error 12.)
Test Case '-[HTTPTests.HTTPClientTests testAmazonWithTLS]' failed (1.096 seconds).
Test Case '-[HTTPTests.HTTPClientTests testExampleCom]' started.
Test Case '-[HTTPTests.HTTPClientTests testExampleCom]' passed (0.633 seconds).
Test Case '-[HTTPTests.HTTPClientTests testGoogleAPIsFCM]' started.
Test Case '-[HTTPTests.HTTPClientTests testGoogleAPIsFCM]' passed (0.122 seconds).
Test Case '-[HTTPTests.HTTPClientTests testGoogleAPIsFCMWithTLS]' started.
[ERROR] [HTTP] [HTTPClient] uncleanShutdown [HTTPClient.swift:32]
<unknown>:0: error: -[HTTPTests.HTTPClientTests testGoogleAPIsFCMWithTLS] : failed: caught error: The operation couldn’t be completed. (NIOOpenSSL.OpenSSLError error 12.)
Test Case '-[HTTPTests.HTTPClientTests testGoogleAPIsFCMWithTLS]' failed (0.134 seconds).
Test Case '-[HTTPTests.HTTPClientTests testHTTPBin418]' started.
Test Case '-[HTTPTests.HTTPClientTests testHTTPBin418]' passed (0.684 seconds).
Test Case '-[HTTPTests.HTTPClientTests testHTTPBinAnything]' started.
Test Case '-[HTTPTests.HTTPClientTests testHTTPBinAnything]' passed (0.661 seconds).
Test Case '-[HTTPTests.HTTPClientTests testHTTPBinRobots]' started.
Test Case '-[HTTPTests.HTTPClientTests testHTTPBinRobots]' passed (0.664 seconds).
Test Case '-[HTTPTests.HTTPClientTests testQuery]' started.
Test Case '-[HTTPTests.HTTPClientTests testQuery]' passed (0.669 seconds).
Test Case '-[HTTPTests.HTTPClientTests testZombo]' started.
Test Case '-[HTTPTests.HTTPClientTests testZombo]' passed (0.769 seconds).
Test Suite 'HTTPClientTests' failed at 2018-05-30 12:47:53.499.
	 Executed 9 tests, with 2 failures (2 unexpected) in 5.433 (5.435) seconds
Test Suite 'HTTPTests.xctest' failed at 2018-05-30 12:47:53.499.
	 Executed 9 tests, with 2 failures (2 unexpected) in 5.433 (5.435) seconds
Test Suite 'Selected tests' failed at 2018-05-30 12:47:53.500.
	 Executed 9 tests, with 2 failures (2 unexpected) in 5.433 (5.436) seconds
Program ended with exit code: 1

Test session log:
	/var/folders/2m/1rc0qztj6qnff96q9bsvrnsm0000gn/T/com.apple.dt.XCTest/IDETestRunSession-AFF26335-9091-44A0-94B0-A1133FD739ED/HTTPTests-7F405052-E380-46E0-B3A4-FDF2E5F91853/Session-HTTPTests-2018-05-30_124746-WxVGKK.log

Please check if you can reduce those errors.

Halimjon Juraev · Answer 7 · Fri Jun 15 2018 00:14:30 GMT+0800 (China Standard Time)

HTTPClient.connect(scheme: .https, hostname: "fcm.googleapis.com", port: 443, on: subContainer) { error in debugPrint(error) }
This little code can generate this error

Tanner · Answer 8 · Fri Jun 22 2018 05:30:11 GMT+0800 (China Standard Time)

So indeed amazon is not sending the close notify packet. Which is technically a violation of SSL, but one that happens quite commonly. The reason behind this is that If the HTTP message includes a clear content-length, then HTTP is capable of detecting a MITM attack trying to close the connection too early. Because HTTP can detect the attack, the SSL close notify isn't as important.

So the proper solution here would be for Vapor to use its knowledge about the state of the HTTP requests to determine whether this error can be ignored. Basing the decision on whether or not there is a pending request seems like it might be a good option.

Luca D'Alberti · Answer 9 · Sun Jan 20 2019 03:57:18 GMT+0800 (China Standard Time)

Hi @tanner0101, this is still happening when reaching, for example, Google search (https://www.google.com/search?q=vapor)

Is there any possible workaround?

Tanner · Answer 10 · Wed Jan 23 2019 02:30:02 GMT+0800 (China Standard Time)

Hi @dalu93, thanks for the poke on this. I've implemented a fix here (#326). However, this is targeting Vapor 4. For Vapor 3, I'd recommend using URLSession (FoundationClient in Vapor). As I mentioned in the previous comment, from what the authors of the underlying transport package say, the error being thrown is actually correct. Google is unfortunately not following TLS best practices.