Smarter logic for when to send window updates
jbrownsw opened this issue · comments
There's already a TODO in window_handler.dart for this, but this just recently bit me. Basically we can get in a state in our application where if we have enough streams going there's not enough CPU available to read from the socket quick enough. The first symptom I noticed was increased memory usage in these cases. Investigating it further I found that I'm also sending a completely excessive amount of data while receiving a stream (20+ Mbps). After doing a quick 10 second wireshark to analyze the data it turned out that 330,654 of 356,707 packets were window_update packets. Given that each window_update packet is 67 bytes this amounts to the receiver sending 22MB worth of data over the network in 10 seconds. Many of these updates are less than 100 bytes and some are even 1 byte. This is clearly not efficient at all.
I'll give a bit more background on the application and use case here. This is a flutter cross platform (desktop and mobile) video streaming application that supports multiple streams of video. We use grpc-dart for communication which obviously uses this library. I've seen this on desktop and mobile. The only difference is mobile generally just takes less streams in order to hit this scenario. In the scenario mentioned above I was running the desktop version and streaming 8 video streams with decoding/display disabled just to rule out anything else using CPU and memory. The intended behavior here when we don't have enough CPU to receive all frames is the server side (our software) should start dropping frames based on how fast we're receiving them to slow down the amount of data being sent. I haven't yet checked to see if that's actually happening, but I suspect even if it were this issue might still prevent that from helping.
Sadly as I was just looking up the version to add here I realized we're using #98 and I looked at the changes and clearly those changes will cause more window_updates, but I don't necessarily think that's the root of the problem. I'm pretty sure those changes are valid and should be happening to keep the window size in sync anyway. I will do more testing without those changes (assuming everything still works without them).
I tried just using the latest release and still ran into the same issue so I don't think #98 is affecting things much here. I made a simple change to window_handler.dart to wait for a minimum of 1KB before sending the window update and saw an immediate drop in everything. CPU usage went down to 1% rather than maxing a core, memory usage remained stable, and the outbound traffic dropped dramatically, but was still significantly higher than I would expect at 2Mbps. I increased the minimum window size update to 2KB and outbound traffic went down to under 1Mbps. I was even able to increase the number of streams without bringing the application to a grinding halt, but I think I'm at a point where the sender can't send enough data over the internet so I'm going to have to modify this test scenario to run locally and see what happens when I can get it to start choking again.
Obviously there's no real magic number that's going to work for everyone, but I think this illustrates a clear need for delaying window size updates so they're sent less frequently and with larger sizes. I know a simple test application to reproduce this would be beneficial so I'll see what I can do after I've done some more of my own testing.
Hi @jbrownsw , did you receive any further response on this or have you thought about a pull request with your fix ? We are definitely seeing a lot of window_update packets streaming data from a Grpc server to a flutter client. Would you mine sharing your fix ?
Sorry, I'm a bit late to respond here. I no longer work on that project anymore, but I just made a fork and changed it. You can see my change here:
exacqvision-client@eb12b37
I just arbitrarily settled on 10KB and it seemed to work fairly well for what we were doing. Your milage may vary.