Http/2 WebSocket streams do not close gracefully

[Chris Ross]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

Http/2 WebSocket streams do not close gracefully with END_STREAM flags, they send/receive RST_STREAM's instead.

the application completed without reading the entire request body.
sending RST_STREAM frame for stream ID 7 with length 4 and flags 0x0.

The client initiates the WebSocket close, the server responds, the server application unwinds and completes the response with and END_STREAM flag. At this point it sees that the client has not sent an END_STREAM flag so the server sends a RST_STREAM message to close the incoming channel.

The client seems to cache this failure and downgrade future WebSocket requests to HTTP/1.1. I've reproduced this with Chrome/Edge and Firefox. To go back to HTTP/2 often requires closing all browsers or switching to incognito.

(Note Firefox requires opting into H2 WebSockets in about:config)

If instead I have the server pause for 5s after sending the WS close message, the client sends a RST_STREAM message to abort the stream.

If instead I have kestrel skip sending the RST_STREAM message after the END_STREAM, the client never sends RST_STREAM or END_STREAM on that request, it seems to time out the connection soon afterwards.

Expected Behavior

The client should be sending us an END_STREAM message immediately after sending the WS Close message. Why doesn't it?

https://datatracker.ietf.org/doc/html/rfc6455#section-7.1.1
To Close the WebSocket Connection, an endpoint closes the
underlying TCP connection. An endpoint SHOULD use a method that
cleanly closes the TCP connection, as well as the TLS session, if
applicable, discarding any trailing bytes that may have been
received. An endpoint MAY close the connection via any means
available when necessary, such as when under attack.

https://datatracker.ietf.org/doc/html/rfc8441#section-5
The HTTP/2 stream closure is also analogous to the TCP connection
closure of [RFC6455]. Orderly TCP-level closures are represented as
END_STREAM flags ([RFC7540], Section 6.1). RST exceptions are
represented with the RST_STREAM frame ([RFC7540], Section 6.4) with
the CANCEL error code ([RFC7540], Section 7).

Steps To Reproduce

Repro app:
https://github.com/dotnet/aspnetcore/blob/146f49fdf09916d0c63e82570bfe059d7fb845e6/src/Middleware/WebSockets/samples/EchoApp/Program.cs

1. Connect the WebSocket using HTTP/2
2. Send a message
3. Close the WebSocket from the client
4. Review the server logs
5. Create a new WebSocket from the client. This is likely to downgrade to HTTP/1.1.

Exceptions (if any)

No response

.NET Version

7.0-preview6

Anything else?

trce: Microsoft.AspNetCore.Server.Kestrel.Http2[37]
      Connection id "0HMIJKKPR7BUG" received HEADERS frame for stream ID 7 with length 154 and flags END_HEADERS, PRIORITY.
info: Microsoft.AspNetCore.Hosting.Diagnostics[1]
      Request starting HTTP/2 CONNECT https://localhost:5001/ - -
dbug: Microsoft.AspNetCore.WebSockets.WebSocketMiddleware[1]
      WebSocket compression negotiation accepted with values 'permessage-deflate; client_max_window_bits=15'.
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[49]
      Connection id "0HMIJKKPR7BUG" sending HEADERS frame for stream ID 7 with length 105 and flags END_HEADERS.
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[37]
      Connection id "0HMIJKKPR7BUG" received DATA frame for stream ID 7 with length 7 and flags NONE.
dbug: EchoApp[0]
      Received Frame Text: Len=0, Fin=True:
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[49]
      Connection id "0HMIJKKPR7BUG" sending DATA frame for stream ID 7 with length 3 and flags NONE.
dbug: EchoApp[0]
      Sent Frame Text: Len=0, Fin=True:
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[37]
      Connection id "0HMIJKKPR7BUG" received WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0.
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[37]
      Connection id "0HMIJKKPR7BUG" received WINDOW_UPDATE frame for stream ID 7 with length 4 and flags 0x0.
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[37]
      Connection id "0HMIJKKPR7BUG" received DATA frame for stream ID 7 with length 27 and flags NONE.
dbug: EchoApp[0]
      Received Frame Close: NormalClosure Closing from client
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[49]
      Connection id "0HMIJKKPR7BUG" sending DATA frame for stream ID 7 with length 23 and flags NONE.
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
      Request finished HTTP/2 CONNECT https://localhost:5001/ - - - 200 - - 6890.5193ms
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[49]
      Connection id "0HMIJKKPR7BUG" sending DATA frame for stream ID 7 with length 0 and flags END_STREAM.
info: Microsoft.AspNetCore.Server.Kestrel[32]
      Connection id "0HMIJKKPR7BUG", Request id "0HMIJKKPR7BUG:00000007": the application completed without reading the entire request body.
trce: Microsoft.AspNetCore.Server.Kestrel.Http2[49]
      Connection id "0HMIJKKPR7BUG" sending RST_STREAM frame for stream ID 7 with length 4 and flags 0x0.

[Chris Ross]

With chrome logs enabled I see one related error.
"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --enable-logging=stderr --v=3
[10772:14408:0629/112726.774:WARNING:spdy_session.cc(3273)] Received RST for invalid stream7
Stream 7 was the WebSocket stream and this RST is the same as the one from Kestrel's logs above.

      Connection id "0HMIJKKPR7BUG", Request id "0HMIJKKPR7BUG:00000007": the application completed without reading the entire request body.
      Connection id "0HMIJKKPR7BUG" sending RST_STREAM frame for stream ID 7 with length 4 and flags 0x0.

Why is this RST_STREAM invalid? The client never sent and END_STREAM flag, the stream is still half-open. It seems like the client discards the stream immediately after END_STREAM, without closing the request half. If I send the RST_STREAM instead of the END_STREAM then I don't get this error, but future WS's still get downgraded.

[Chris Ross]

The client seems to cache this failure and downgrade future WebSocket requests to HTTP/1.1. I've reproduced this with Chrome/Edge and Firefox. To go back to HTTP/2 often requires closing all browsers or switching to incognito.

I've found an explanation for this part.

https://docs.google.com/document/d/1ZxaHz4j2BDMa1aI5CQHMjtFI3UxGT459pjYv4To9rFY/edit

HttpStreamFactoryImpl::Job tries to reuse an existing SpdySession (representing an HTTP/2 connection) where the server has already advertised WebSocket support. If no such connection exists, it creates a non-HTTP/2 connection, just like before.

It's not caching a failure, it's just defaulting to Http/1.1 if there isn't an Http/2 connection at the time. A simple refresh isn't sufficient since the HTML is heavily cached. A forced refresh (shift+F5) re-establishes the HTTP/2 connection and reloads the HTML. After that HTTP/2 WebSockets work again. I don't see this issue with Firefox, it seems to cache across connections that H/2 WS are supported.

That doesn't explain the original issue of why the client doesn't cleanly close the stream, but it makes it less of a concern.