taskrabbit / empujar

When you need to push data around, you push it. A node.js ETL tool.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Support multiple processes

evantahler opened this issue · comments

Empujar's async workflow can be enhanced to execute across more than one process.

How's the multiple processes support so far ?

No progress what-so-ever!

Want to help out?

The main choice to make is:

  • do we use node's cluster module / IPC channels to communicate to sub-processes
  • do we use something like redis to share state over, and a polling/worker model

option 1 is simpler, but option 2 enables us to use more than one computer not just more processes/cores.

We would also need to use setters/getters for data stored at the book and chapter level, rather than modifying that object directly, as data would need to be persisted/shared between all processes.