yield ordered items that finish in order immediately
vitamins opened this issue · comments
For reference the description of ordered from the docs:
Creates a stage that sorts its elements based on their order of creation on the source iterable(s) of the pipeline. This stage will not yield until it accumulates all of the elements from the previous stage, use this only if all elements fit in memory.
It would be nice if ordered would yield items that finish in order immediately.
I've thought about this and having it yield in a smarter way would only work well for when you only have map
s (no flat_map
or filter
), but given map-only is pretty common it might be worth it.
I'm struggling to implement the same mechnism here, so would be nice to have it in the API 👍
I will be asking on StackOverflow in the meantime.
Edit: I asked on SO:
https://stackoverflow.com/questions/67426570/ordered-iterator-with-pypeln-map-stage-results