Issue with race of cancelling data on client side and sending data on server side
mkustermann opened this issue · comments
If the client side cancels a stream and the server side sends more data at the same time, the client reacts incorrectly to the incoming data frames.
From grpc/grpc-dart#131 (comment):
With the following events:
- client calls a method that streams results
- server starts the stream and send the first DataFrame
- client receives the first DataFrame
- client cancels the stream
- server (that has not yet received the cancellation) sends the second DataFrame
- server receives cancellation and stops sending DataFrame
- client receives the second DataFrame
The last line leads today to an error (grabbed with some additionnal logs):
Connection._catchProtocolErrors: Bad state: Was in the process of closing.
#0 _StreamMessageQueueIn&Object&TerminatableMixin&ClosableMixin.ensureNotClosingSync (package:http2/src/error_handler.dart:73:7)
#1 StreamMessageQueueIn.enqueueMessage (package:http2/src/flowcontrol/stream_queues.dart:217:5)
#2 ConnectionMessageQueueIn._tryDispatch (package:http2/src/flowcontrol/connection_queues.dart:317:10)
#3 ConnectionMessageQueueIn._addMessage (package:http2/src/flowcontrol/connection_queues.dart:295:5)
#4 ConnectionMessageQueueIn.processDataFrame (package:http2/src/flowcontrol/connection_queues.dart:253:5)
#5 StreamHandler._handleDataFrame (package:http2/src/streams/stream_handler.dart:634:19)
#6 StreamHandler._processStreamFrameInternal.<anonymous closure> (package:http2/src/streams/stream_handler.dart:596:11)
#7 _StreamHandler&Object&TerminatableMixin.ensureNotTerminatedSync (package:http2/src/error_handler.dart:33:13)
#8 StreamHandler._processStreamFrameInternal (package:http2/src/streams/stream_handler.dart:509:12)
#9 StreamHandler.processStreamFrame (package:http2/src/streams/stream_handler.dart:469:7)
#10 Connection._handleFrameImpl (package:http2/src/connection.dart:368:16)
#11 Connection._setupConnection.<anonymous closure>.<anonymous closure> (package:http2/src/connection.dart:155:34)
#12 Connection._catchProtocolErrors (package:http2/src/connection.dart:300:9)
#13 Connection._setupConnection.<anonymous closure> (package:http2/src/connection.dart:155:7)
#14 StackZoneSpecification._registerUnaryCallback.<anonymous closure>.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:129:26)
#15 StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:209:15)
#16 StackZoneSpecification._registerUnaryCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:129:14)
#17 _rootRunUnary (dart:async/zone.dart:1132:38)
#18 _CustomZone.runUnary (dart:async/zone.dart:1029:19)
#19 _CustomZone.runUnaryGuarded (dart:async/zone.dart:931:7)
#20 _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:336:11)
#21 _DelayedData.perform (dart:async/stream_impl.dart:584:14)
#22 _StreamImplEvents.handleNext (dart:async/stream_impl.dart:700:11)
#23 _PendingEvents.schedule.<anonymous closure> (dart:async/stream_impl.dart:660:7)
#24 StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:209:15)
#25 StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:119:48)
#26 _rootRun (dart:async/zone.dart:1120:38)
#27 _CustomZone.run (dart:async/zone.dart:1021:19)
#28 _CustomZone.runGuarded (dart:async/zone.dart:923:7)
#29 _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:963:23)
#30 StackZoneSpecification._run (package:stack_trace/src/stack_zone_specification.dart:209:15)
#31 StackZoneSpecification._registerCallback.<anonymous closure> (package:stack_trace/src/stack_zone_specification.dart:119:48)
#32 _rootRun (dart:async/zone.dart:1124:13)
#33 _CustomZone.run (dart:async/zone.dart:1021:19)
#34 _CustomZone.runGuarded (dart:async/zone.dart:923:7)
#35 _CustomZone.bindCallbackGuarded.<anonymous closure> (dart:async/zone.dart:963:23)
#36 _microtaskLoop (dart:async/schedule_microtask.dart:41:21)
#37 _startMicrotaskLoop (dart:async/schedule_microtask.dart:50:5)
#38 _runPendingImmediateCallback (dart:isolate/runtime/libisolate_patch.dart:113:13)
#39 _RawReceivePortImpl._handleMessage (dart:isolate/runtime/libisolate_patch.dart:166:5)
This leads to connection (and all its streams) closing.
IMO this second DataFrame shoud be discard. In the http2 spec I see:
However, after sending the RST_STREAM, the sending endpoint MUST be prepared to receive and
process additional frames sent on the stream that might have been sent by the peer prior to the arrival
of the RST_STREAM.
From grpc/grpc-dart#131 (comment):
There are still issue with the removal of the stream from StreamHandler._openStreams by streams_handler.dart . Once called the next incoming frame doesn't match any open stream and this leads to a protocol error. It looks like cancelled streams should also be tracked in this class.
Here's the simplest repro I found:
// --- server.dart
import 'dart:io';
import 'package:http2/transport.dart';
main() async {
final serverSocket = await ServerSocket.bind(InternetAddress.anyIPv4, 9896);
await for (final socket in serverSocket) {
final connection = new ServerTransportConnection.viaSocket(socket);
await for (final stream in connection.incomingStreams) {
await stream.incomingMessages.toList();
stream.outgoingMessages
..add(new DataStreamMessage([1]))
..add(new DataStreamMessage([2]))
..close();
}
}
}
// --- client.dart
import 'dart:io';
import 'package:http2/transport.dart';
main() async {
var socket = await Socket.connect('localhost', 9896);
var transport = new ClientTransportConnection.viaSocket(socket);
var stream = transport.makeRequest([], endStream: true);
await for (var message in stream.incomingMessages) {
if (message is DataStreamMessage) {
break; // this cancels the incoming stream
}
}
assert(transport.isOpen); // fails but should be ok
await transport.finish();
}