rvagg / node-worker-farm

Distribute processing tasks to child processes with an über-simple API and baked-in durability & custom concurrency options.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

workerFarm.end is not working

Kequc opened this issue · comments

commented

I'm using node 10.0.0 on macOS.

When I call workerFarm.end it immediately terminates my running processes which I understand is not supposed to happen, and then they all start again. Possibly because they are retrying, but regardless, the queue picks up again and keeps going.

const workerFarm = require('worker-farm');

const options = {
    maxConcurrentCallsPerWorker: 1
};

// This worker spawns a mongodb client inside it
// But that shouldn't be relevant since node-worker-farm sends SIGKILL
// to close the processes.
const queue = workerFarm(options, require.resolve('./worker.js'));

// While my script is running I press CTRL-C to terminate the process
// This is where things go wrong because the processes all immediately exit and
// start up again.
process.on('SIGINT', () => {
    console.info('\rGracefully shutting down...');
    workerFarm.end(queue, finish);
});

// this is close to 300,000 items
const arr = [];

for (const item of arr) {
    queue(item, onProcessed);
}

let counter = 0;

function onProcessed() {
    counter++;
    if (counter === arr.length) {
        workerFarm.end(queue, finish);
    }
}

function finish() {
    console.info('Done');
}
commented

Tried node 8.9.3 with the same problem.

commented

Seems that SIGINT must be having some effect. When I replace my listener with a timeout, my child processes don't stop. If you have any hints on how to fix that I'd be very happy to know, as gracefully shutting down on SIGINT was my goal.

However it seems the queue will continue to run with this configuration.

Expected behaviour:

The active queue items would finish and then the workerFarm.end callback would run. Queue items not currently being processed would be discarded.

Sorry for the very delayed response but unfortunately I don't have answers for this. I'd expect it to behave the way you do too. Either there's something about SIGINT and its interaction with child processes or there's something specific about your child.
I'm assuming this is a stale issue for you given it's nearly a year old so I'm closing. If you want to discuss further maybe you could provide a usable minimal test case that triggers this behaviour?

i am also seeing this :(