cgarciae / pypeln

Concurrent data pipelines in Python >>>

Home Page:https://cgarciae.github.io/pypeln

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

yield ordered items that finish in order immediately

vitamins opened this issue · comments

For reference the description of ordered from the docs:
Creates a stage that sorts its elements based on their order of creation on the source iterable(s) of the pipeline. This stage will not yield until it accumulates all of the elements from the previous stage, use this only if all elements fit in memory.
It would be nice if ordered would yield items that finish in order immediately.

I've thought about this and having it yield in a smarter way would only work well for when you only have maps (no flat_map or filter), but given map-only is pretty common it might be worth it.

I'm struggling to implement the same mechnism here, so would be nice to have it in the API 👍

I will be asking on StackOverflow in the meantime.
Edit: I asked on SO:
https://stackoverflow.com/questions/67426570/ordered-iterator-with-pypeln-map-stage-results