GoawayFrame puts http2 connection in a bad state

[Todd Volkert]

Spawned from dart-lang/appengine#117

package:appengine registers a single gcloud:DatastoreDB instance upon server startup, which triggers the following objects to be persisted as part of its object graph:

• ↳ gcloud:DatastoreDB maintains a reference to a gcloud:Datastore
  • (implemented by appengine:GrpcDatastoreImpl)
• ↳ appengine:GrpcDatastoreImpl maintains a reference to grpc:ClientChannel
• ↳ grpc:ClientChannel maintains a reference to grpc:Http2ClientConnection
• ↳ grpc:Http2ClientConnection maintains a reference to http2:ClientTransportConnection

I'm seeing cases where the following code is hitting:

http2/lib/src/connection.dart

Lines 354 to 356 in 0484372

 } else if (frame is GoawayFrame) { 
   _streams.processGoawayFrame(frame); 
   _finishing(active: false); 

This causes the connection's state to get marked as IsFinishing/PassiveFinishing. Then later, grpc tries to issue another request:

https://github.com/grpc/grpc-dart/blob/1213bc546b46da385afc94e645dd44fefcfb7f82/lib/src/client/http2_connection.dart#L130

And it results in the following error:

http2/lib/src/connection.dart

Lines 461 to 463 in 0484372

 throw new StateError( 
     'The http/2 connection is finishing and can therefore not be used to ' 
     'make new streams.'); 

Since the underlying http2 connection is retained by our server-wide connection to the datastore, every single subsequent HTTP request to my App Engine server that tries to talk to the datastore fails with this error ("gRPC Error (14, Error making call: Bad state: The http/2 connection is finishing and can therefore not be used to make new streams")

@jonasfj @mkustermann @sigurdm @a14n

[Jonas Finnemann Jensen]

@sigurdm, maybe we can step through this sometime Monday/Tuesday, it feels like a failure to reconnect - in grpc, not sure why. I thought we had checked that it would reconnect :)

@tvolkert, is it all subsequent requests, or only the ones happening concurrently to the request that failed?
Does the connection continue to be broken minutes later too?

[Todd Volkert]

@jonasfj it is all subsequent requests - I still saw it happening for all requests 12 minutes after first occurrence, and then I switched traffic back over to the Go backend.

[Martin Kustermann]

Generally speaking http/2 connections can go into finishing state for various reasons and you cannot make new streams on such a connection. So this situation has to be handled by the user of package:http2, namely the grpc implementation.

The original grpc implementation in package:appengine/src/grpc_api_impl/grpc.dart handled this just fine.

Please see my comment on the original issue dart-lang/appengine#117 (comment).

[Todd Volkert]

Thanks Martin! Third time's a charm in filing this issue in the correct repo 😄 -- I'm gonna close this an consolidate all comments into grpc/grpc-dart#228