A simple way to create a Node.JS Transform stream which processes in parallel. You can limit the concurrency (default is 16) and order is not preserved (so chunks/objects can end up in a different order to the order they started in if the transform functions take different amounts of time).
Build using through2 and has the
same API with the exception of an extra maxConcurrency
option.
Non-objectMode
streams are supported for completeness but I'm not
sure they'd be useful for anything.
Written by Thomas Parslow (almostobsolete.net and tomparslow.co.uk) as part of Active Inbox (activeinboxhq.com).
npm install --save through2-concurrent
Process lines from a CSV in paralel. The order the results end up in the "all" is not deterministic.
var through2Concurrent = require('through2-concurrent');
var all = [];
fs.createReadStream('data.csv')
.pipe(csv2())
.pipe(through2Concurrent.obj(
{maxConcurrency: 10},
function (chunk, enc, callback) {
var self = this;
someThingAsync(chunk, function (newChunk) {
self.push(newChunk);
});
}))
.on('data', function (data) {
all.push(data)
})
.on('end', function () {
doSomethingSpecial(all)
})
Fixed or improved stuff? Great! Send me a pull request through GitHub or get in touch on Twitter @almostobsolete or email at tom@almostobsolete.net