Errors.CloseTimeout (Close operation timed out)

[itler90]

Hi,
I am new to your library and (sometimes/sporadically) I get an error. The error refers to the close method, which I want to use to wait for all running tasks AND queued tasks were finished. But as far as I read, there is no way to wait also for queued tasks, is there?

Back to my issue. Here are the error message and my two js files just to test the library. I think the error is ok, if close is called while are no queued tasks. Therefore I added a while-loop before calling close to check whether there are queued tasks.

Can you just support me please and tell me what is my mistake (probably this is no bug in the library but my fault)?

Thank you!

Error message

Exception has occurred: Error: Close operation timed out
  at Object.CloseTimeout (<myPath>\node_modules\piscina\dist\index.js:72:25)
    at throwOnTimeOut (<myPath>\node_modules\piscina\dist\index.js:727:26)
    at async ThreadPool.close (<myPath>\node_modules\piscina\dist\index.js:730:13)
    at async file:///<myPath>/main.js:29:1

main.js

import * as os from 'node:os';
import { Piscina } from 'piscina';

const piscina = new Piscina({
  filename: 'worker.js',
  maxThreads: os.availableParallelism()
});

for(let i = 0; i < 20; i++)
{
  piscina.run(i).then(([x, result]) => {
    console.log("Thread: " + x + "; Result: " + result);
  });
}

function Sleep(milliseconds) {
  return new Promise(resolve => setTimeout(resolve, milliseconds));
}

// here I just want to ensure that all queued tasks running and there are no queued tasks
// since the close method below waits just for running tasks to be finished
while (piscina.queueSize)
{
  await Sleep(10);
}

await piscina.close();

process.exit();

worker.js

// just a nonsensical task for the CPU to test piscina
export default (num) => {
  let rv = 0;
  for (let x = 0; x < 1000000; x++)
  {
    rv += Math.pow(Math.sqrt(num), 512);
  }

  return [num, rv];
};

[Carlos Fuentes]

Hey!

First, yes, as you already said the erro is working as expected; the close method won't wait for the queue to be flushed but rather that all running tasks are done before closing. Tho, the event is quite interesting because it might mean that something is causing the worker to report back the state in a delayed manner.

Second, not sure what your main goal is, but if you are seeking to see when its time to add more tasks to the queue, I'd definitely suggest you to use the drain event from Piscina.
e.g.

const { once } = require('node:event');

// Piscina code
await once(piscina, 'drain');

This event is triggered when the pool has room capacity for adding more tasks.

What I acknowledge is that we do not have an event for when pool and its task queue is totally empty, maybe adding something like idle event can be of support.

[itler90]

Hi @metcoder95,

Thanks for your quick response!
I already saw the drain event but my main goal is to wait for all elements are executed completely. Now I just implemented a workaround:
I introduced a global variable finished as counter. Within the task argument of the piscina.run call I increment the counter. So I'm sure that each task has been finished and after that the counter is incremented. Next to the for-loop of the piscina.run calls I added following (with maxLength == 20 in my example above):

while (finished != maxLength)
{
  await Sleep(100);
}

So every 100 ms I check whether my counter reached my max element size and thus I can detect that all tasks have been executed completely.

For me this issue is closed. Thanks again for your response! 😄