OutboundFlowControlBuffer probably allocates unnecessarily

[Cory Benfield]

OutboundFlowControlBuffer defines a data type called DataBuffer, which is essentially a wrapper around MarkedCircularBuffer. The purpose of this DataBuffer is to store the pending flow-controlled frames, as well as any frames that need to be totally ordered with those flow controlled frames (namely headers).

This data structure tries to aim for simplicity by defining typealias BufferElement = (HTTP2Frame.FramePayload, EventLoopPromise<Void>?). This seems sensible, except that HTTP2Frame.FramePayload has a number of indirect cases. These cases force a heap-allocation when they are used. They were added in #110 to reduce the cost of passing a HTTP2Frame around through the ChannelPipeline, and did so by shrinking the size of the enum.

Unfortunately for us, OutboundFlowControlBuffer buffers only two types of frame payload: data and headers. These are both indirect cases, meaning that when we go to buffer them we inadvertently trigger a new heap allocation!

The way we need to avoid this is a bit subtle. If we take the example of the .data path, the code right now looks like this:

swift-nio-http2/Sources/NIOHTTP2/Frame Buffers/OutboundFlowControlBuffer.swift

Lines 45 to 53 in 94c35c6

         switch frame.payload { 
         case .data(let body): 
             // We buffer DATA frames. 
             if !self.streamDataBuffers[frame.streamID].apply({ $0.dataBuffer.bufferWrite((.data(body), promise)) }) { 
                 // We don't have this stream ID. This is an internal error, but we won't precondition on it as 
                 // it can happen due to channel handler misconfiguration or other weirdness. We'll just complain. 
                 throw NIOHTTP2Errors.noSuchStream(streamID: frame.streamID) 
             } 
             return .nothing 

What needs to happen is that, instead of buffering .data(body), we need to buffer frame.payload. This will pass the pre-CoW-boxed data along, and should shrink allocations on that path. Naturally, we do need to verify that we observe the expected improvement.