Is there a programmatic way to kill all workers of a queue?
0xgeert opened this issue · comments
During normal operations I'm using idleTimeout > 0
to allow workers to be kept around, allowing for some persistent caching in between invocations.
This has worked well for me except I'm now occasionally getting a OOM in a workerthread. Ideally I would get the source of the OOM, but the more pragmatic way seems to once in a while 'just' kill the workerthreads, requiring Piscina to create a new workerthread instance.
Is there a way to give this command to a Piscina instance? Alternatively, if that's not doable, I could periodically call destroy()
and create a new Piscina instance I guess
👋
Currently, there's no way to achieve that in Piscina through its API.
Depending on how your workload looks like, you can periodically call Piscina#close
to flush all the pending tasks and later destroy the pool.
Although destroyed, Piscina
does not hold this state, so you can perfectly close/destroy the pool and start adding tasks right after without a problem.
e.g.
const { join } = require('node:path')
const Piscina = require('.')
const pool = new Piscina({
maxThreads: 2,
minThreads: 2,
filename: join(__dirname, 'worker.js')
})
;(async () => {
for (let i = 0; i < 10; i++) {
await pool.run(`console.log(${i})`)
}
await pool.close()
// or alternatively await pool.destroy()
for (let i = 0; i < 10; i++) {
pool.run(`console.log(${i})`)
}
})()
So just to confirm: pool.close()
should be enough to close all worker threads completely (thus freeing up all memory from worker threads)? Then afterwards I can just do pool.run()
to add workers again?
Then I would not need pool.destroy()
.
Or asked differently, is there any reason to use pool.destroy()
at all in the above scenario?
Use pool.destroy
if you want to fully tear down the threads independently of their state (will discard any actual work and mark them as aborted
).
Use pool.close
if you seek for a graceful shutdown 👍