Potential issue with `put_unique`
Neverlord opened this issue · comments
Reported in the testing group: it seems like there's a potential performance issue with put_unique. Many writes from Zeek lead to an increasing memory usage over time until the system keels over.
To clarify, the actual report was about memory bloat. So far, I couldn't reproduce the bloat itself. What I did find is that Broker seems to do fine for throughput, but latency is quite high. This makes roundtrips for put_unique quite slow and Broker can only do ~4-5k put_unique per second with one writer/clone. Adding more writers/clones slows down further. I didn't try "interleaving" writes, though. The setup in the testing groups seems to be calling put_unique non-stop (without waiting for results first). That could explain the bloat if the system simply does more writes than Broker can handle.