andersea / slurry

Python async data processing microframework

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

The broadcast method is unsafe

andersea opened this issue · comments

The way broadcasting to taps and extensions works currently is not safe. It should support backpressure, but currently messages just pile up in a big queue, leading to possible memory leaks and other unpleasant issues. This needs a thorough rethink. There is a lot of discussion available on the issue of broadcasting in the trio community as well.

See for instance: python-trio/trio#987

Naive attemp to make broadcasting safer has been added in the safe-broadcasting branch. It simply wraps the send loop in a separate nursery, which will have the effect of not allowing the source iteration to continue until all sends have completed.