Giters

czheo / syntax_sugar_python

A library adding some anti-Pythonic syntatic sugar to Python

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Is streaming source data possible?

fenom opened this issue · comments

commented

Is streaming from, say, a database cursor possible? I don't want to have to load everything before data starts moving through the pipe.

commented

Surely, pipe is lazily evaluated.

commented

syntax_sugar_python/syntax_sugar/_pipe.py

Line 67 in c680002

return p.map(fn, data)

I found that I had to change this line to use imap and list, just like in multigreenthread, to get my desired behavior. map consumes the entire database cursor before running the function, and I couldn't afford that. My function writes back to the database and discards the data, so even though the IMapIterator was consumed immediately anyway, it was much better than loading all the data first. And I wonder if it's possible to make parallel processing fully lazy by yielding from the IMapIterator and making generator syntax with parentheses, that way the parallel processing stage doesn't have to be the final stage. Square brackets can be kept as eager syntax, just like list comprehension.