buffer pools can be leaky
eric-millin opened this issue · comments
Re pool use here (may also be elsewhere): https://github.com/IBM/fluent-forward-go/blob/main/fluent/protocol/transport.go#L56-L58
There is an edge case bug where the use of sync.Pool can effectively leak memory. Pools are helpful when they contain objects of similar size, but they can be problematic if they contain large outliers. E.g., if 99% of the time a long-running application needs buffers of size 1k, but 1% of the time it needs 1GB, then as the buffers are re-used, they could all eventually be sized to 1GB.
It looks like there are efforts to address this in the stdlib, but it's something you will want to keep an eye on. If you want to get ahead of the issue there are a number of solutions suggested and linked in a thread on the golang repo. Here's a simple one: golang/go#27735 (comment)