ummon-server / ummon-server

Ummon is node.js application for creating, queuing, running and monitoring externally running tasks

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Child processes fail if master fails. This is not ideal.

mattmcmanus opened this issue · comments

As it stands right now, spawned process will terminiate the instant the mast process does. This is good in some ways but troublesome in many more.

spawn supports detached processes but it introduces a number of questions and concerns. Here is an example from http://nodejs.org/docs/latest/api/child_process.html#child_process_child_process_spawn_command_args_options

var fs = require('fs'),
     spawn = require('child_process').spawn,
     out = fs.openSync('./out.log', 'a'),
     err = fs.openSync('./out.log', 'a');

 var child = spawn('prg', [], {
   detached: true,
   stdio: [ 'ignore', out, err ]
 });

 child.unref();

Challenges & Questions that need answering

  • Does each worker have it's own file or do all workers go to the same?
  • How is the file then parsed by ummon? How can useful meta data be added by bunyan?
  • Is there a better way to go about this? Do we want to keep processes running if master fails? Probably.
  • This makes my brain hurt

Here is a stack overflow article about a possible way to reattach running process: http://stackoverflow.com/questions/10112827/reattaching-to-spawned-process-via-nodejs

I don't see how the management of processes can continue through a crash when the master holds so much rapidly-changing state. Are there any examples of other projects where workers survive a master failure?

You're right. It's definitely clear that trying to reattach processes is silly. What I'm thinking about now is should we set it up so that child processes don't bail immediately and finish out if master fails. I have some tests that I want to try with this.