Maximum workload a worker can work on?

[Samundra Khatri]

Hello,
I have a requirement where I need to do some heavy I/O so I created a simple module thekdar
which creates worker (child_process) on basis of workload and a worker can do 10 tasks(configurable) at a time, but now I came to know about napa.js and thinking of giving it a try.
So do napa workers have any work limit? like a number of tasks a worker can do or amount of memory, a worker can consume, also any other way to communicate with the main process? also, can we know which worker is working and which are staying idle?
Sorry, if I asked it at wrong place :)

[Yulong Wang]

Hi @samundrak , following are instant answers to your questions-
So do napa workers have any work limit? like a number of tasks a worker can do or amount of memory, a worker can consume

• There is no limit for the number of tasks for workers/zones, but if you produce tasks faster than consume, there will be one day when you run out of memory.
• There is no limit for memory usage for workers/zones.

also any other way to communicate with the main process?

• You can use node zone to communicate with the main process.

also, can we know which worker is working and which are staying idle?

• No. Actually this information is kept private in zone's scheduler implementation. tasks are always created on a zone, not a specific worker in that zone.
• And no for zone. now there is no public API to retrieve the workload information of a zone.

As you see, I guess you want to monitor/manage/control threads, but it is more complex than monitoring processes. Some of those numbers may even be considered invalid, such as calculating memory usage of a thread. But still, I think in generally diagnosis information is useful. Could you please share me some more detail about what you are trying to do? I am trying to understand the core of the problem.

[Samundra Khatri]

Hello,
My requirement is user will send a request for a task which is to transfer files from one place to another and network is required to do so.
we can get many tasks to do at the same time so we used node child process and manage it by our module. so I thought napajs could help us on this task but I came to know that no native node API is supported on napajs worker.

[Yulong Wang]

I would try to figure out where the bottleneck is.

If it turned out to be the bottleneck is in JavaScript tasks, for example, spending too much time in a synchronized function or code blocks, it's the time for Napa to help. This may happen when you do tasks like map building, string parsing, rendering or searching object, especially when those happens in a big loop.

If there isn't any heavy CPU-bounded tasks, it is more likely the bottleneck is in I/O. In general, Napa does not help much on scenarios of heavy I/O bounded tasks. Node is designed and optimized exactly for this kind of scenarios. Try other options like increasing uv threadpool's size or multi process node.