Working with node-cluster

[Lucas Hamie]

I'm triying to use a Collector with node cluster, but when i use with it, i lose packets.

Say i receive 1k packets on my 1-process server, then i'll receive about 800 in a 2-processes server, 700 in a 3-processes server, and so on.

I started googling a bit, and found that udp4 clustering isn't supported in node 0.10.x as stated on this SO question.

I'm using node v0.10.36 (On ubuntu 14.04), so the first strange thing is that it's working on this version of node. Then it comes this problem of losing packets, which i can't find a solution.

Also tried node v0.11.x but it showed the same results.

Basically, this is my code:

var cluster = require('cluster');
cluster.schedulingPolicy = cluster.SCHED_NONE;

if (cluster.isMaster) {
..
  for (var i = 0; i < 4; i++) {
    cluster.fork();
  }

} else {
  var Collector = require('node-netflowv9');
 Collector({
    port: 9995
  }).on('data', function(packet) {
    totalPacketCount += packet.flows.length;
});
//then print that count.

Any help would be appreciated.

Thanks!

[Delian Delchev]

I don't know, but somehow I don't believe the problem is related to the
nodejs cluster. It may be related to the default kernel read udp buffer.

Could you increase the buffer (take a look here
http://docs.oracle.com/cd/E18930_01/html/821-2431/abeis.html ) and restart
your code and test again?

On Mon, Feb 2, 2015, 21:58 Lucas Hamie [e-mail] wrote:

I'm triying to use a Collector with node cluster, but when i use with it,
i lose packets.

Say i receive 1k packets on my 1-process server, then i'll receive about
800 in a 2-processes server, 700 in a 3-processes server, and so on.

I started googling a bit, and found that udp4 clustering isn't supported
in node 0.10.x as stated on this SO question
http://stackoverflow.com/questions/16724216/node-js-0-10-7-cluster-support-for-udp-dgram
.

I'm using node v0.10.36 (On ubuntu 14.04), so the first strange thing is
that it's working on this version of node. Then it comes this problem of
losing packets, which i can't find a solution.

Also tried node v0.11.x but it showed the same results.

Basically, this is my code:

var cluster = require('cluster');
cluster.schedulingPolicy = cluster.SCHED_NONE;

if (cluster.isMaster) {
..
for (var i = 0; i < 4; i++) {
cluster.fork();
}

} else {
var Collector = require('node-netflowv9');
Collector({
port: 9995
}).on('data', function(packet) {
totalPacketCount += packet.flows.length;
});
//then print that count.

Any help would be appreciated.

Thanks!

—
Reply to this email directly or view it on GitHub
#5.

[Lucas Hamie]

The server doesn't seem to have been dropping packets, i checked it as it is explained in that document using netstat -su, so there should be no reason to increment the buffer size.

Anyways, just tried incrementing the buffer size and I got the same result. I think it has something to do with the cluster module and the sharing of the socket.

Update: To determine that i'm having this problem, I'm using a static sized .pcap file which I inject using tcpreplay, so i always receive the same amount of packets and flows. Using this I find that I'm loosing packets when using the cluster module.

[Delian Delchev]

Hm, that sounds very strange. I haven't seen it before, but to be fair I
haven't tested it yet with cluster. I have to try it myself

Delian

On Tue, Feb 3, 2015 at 3:14 PM, Lucas Hamie [e-mail]
wrote:

The server doesn't seem to have been dropping packets, i checked it as it
is explained in that document using netstat -su, so there should be no
reason to increment the buffer size.

Anyways, just tried incrementing the buffer size and I got the same
result. I think it has something to do with the cluster module and the
sharing of the socket.

—
Reply to this email directly or view it on GitHub
#5 (comment).

[Lucas Hamie]

I'm providing my test code, so you can try it yourself :)

It counts packets per second, and the total amount of packets since server start. To test it, I start a server with numThreads = 1, send a static amount of flows to it, then CTRL+C it, watch the final counter; change numThreads to a larger number, send the same amount of flows to it, and watch that the counter has a different final value.

There it goes:

var cluster = require('cluster');
cluster.schedulingPolicy = cluster.SCHED_NONE;

if (cluster.isMaster) {
  var numCPUs = require('os').cpus().length;
  var numThreads = 1; // numCPUs - 2;
  //Fork the workers, one per CPU
  console.log('Running on a machine with ' + numCPUs + ' cores. Starting ' + numThreads + ' threads.');
  for (var i = 0; i < numThreads; i++) {
    cluster.fork();
  }
  cluster.on('exit', function(deadWorker, code, signal) {
    // The worker is dead.

    var worker = cluster.fork();
    var newPID = worker.process.pid;
    var oldPID = deadWorker.process.pid;

    console.log('Worker with PID ' + oldPID + ' died.');
    console.log('Worker with PID ' + newPID + ' born.');
  });
} else {

  ///////////////////
  /* PROCESS START */
  ///////////////////

  var Collector = require('node-netflowv9');
  var _ = require('lodash');
  var fs = require('fs');
  var inet = require("inet");

  //Initialize counters
  var lastSecondPacketCount = 0;


  var totalPacketCount = 0;


  Collector({
    port: 9995
  }).on('data', function(packet) {
    lastSecondPacketCount += packet.flows.length;
  });

  //Print packets per second
  var interval = setInterval(function() {
    if (lastSecondPacketCount > 0) {
      console.log("Received " + lastSecondPacketCount + " flows this second.");
    }

    //sum totals
    totalPacketCount += lastSecondPacketCount;

    //reset temporals
    lastSecondPacketCount = 0;

  }, 1000);
}

// Printing final counters
process.on('SIGINT', function() {
  if (cluster.isMaster) {} else {
    console.log("Worker stats:" + totalPacketCount);
  }
  process.exit();
});

[Lucas Hamie]

This works fine Delian, you can close this.

[Delian Delchev]

Sorry for the extreme delay. Could you tell me which version of the Node.JS are you using? And which OS? I see that node.js handle in different way the sockets when they are in a Cluster?

[Delian Delchev]

Did you changed anything? As I have been unable to reproduce it, but I see a lot of cluster-event related bugs in the different Node.js versions

[Lucas Hamie]

Nope, just a bug in my testing. It totally works :). Ubuntu 14.04 node 0.10.X