Many Connection Attempts

[motosota]

Using node v0.12.0 & modesl 1.1.2

Something caused the generation of multiple connection attempts per second which syrocketed FS CPU and memory utilisation.

Given that the module doesnt re-connect automatically ( #34 ) I wrote re-connections into my script as follows:

var eslWaitTime = 60000;
function fsConnect() {
  eslConn = new esl.Connection(server, port, password)
    .on("error", function (error) {
      console.log('ESL Connection Error ' + JSON.stringify(error));
      setTimeout(fsConnect, eslWaitTime);
    }).on("esl::end", function () {
      console.log('ESL Connection Ended');
      setTimeout(fsConnect, eslWaitTime);
    }).on("esl::ready", function () {
      // Subscribe to events
      eslConn.events('json' , 'ALL', function() {
        console.log('ESL ready - subscribed to receive events.');
      });
    }).on("esl::event::**", function (event, headers, body) {
      // do stuff
    });
}

eslWaitTime works as expected in controlled circumstances (e.g. FS not running)

The messages I saw when it all started to go wrong:

2015-06-18T11:36:55.098Z  ESL Connection Error {} 
2015-06-18T11:36:55.109Z  ESL Connection Error {} 
2015-06-18T11:36:55.112Z  ESL Connection Error {} 
2015-06-18T11:36:55.116Z  ESL Connection Error {} 
2015-06-18T11:36:55.124Z  ESL Connection Error {} 
2015-06-18T11:36:55.126Z  ESL Connection Error {} 
2015-06-18T11:36:55.130Z  ESL Connection Error {} 
2015-06-18T11:36:55.133Z  ESL Connection Error {} 
2015-06-18T11:36:55.141Z  ESL Connection Error {} 
.. lots more ..
.. some time later ..
2015-06-18T11:37:55.293Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.308Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.322Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.330Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.338Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.347Z ESL ready - subscribed to receive events. 
2015-06-18T11:37:55.386Z ESL ready - subscribed to receive events. 
.. patterns repeat and then start getting
2015-06-18T11:39:03.590Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}
2015-06-18T11:39:03.592Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}
2015-06-18T11:39:03.593Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}
2015-06-18T11:39:03.593Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}
2015-06-18T11:39:03.597Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}
2015-06-18T11:39:03.598Z ESL Connection Error {"code":"ETIMEDOUT","errno":"ETIMEDOUT","syscall":"connect"}

So given the module doesn't reconnect, and my script should only reconnect every 60 seconds - any idea what might have caused this & how to fix?

The problem persisted until I restarted the module.

Cheers

Mike

[Chad Engler]

Hm, I'm confused why your first logs are just blank. There shouldn't be any instance of an error event with no error object.

Do you get these same issues on node 0.10? I have never run this modules on 0.12 before.

[motosota]

Yes that had me confused too (as well as the ms level reconnection attempts).

I've been developing the app for several weeks against a test FS & running it with data from a live FS for several more.
In that time this is the first time I've seen the problem, so even if I downgraded to 0.10 I've no idea how to try to even trigger the problem other than sit and wait.

I've checked syslogs on both servers and no unusual messages from around the time the problem started - just apps complaining of out of memory some time later.

Scratches head.

[Chad Engler]

Best case would be to open it in a debugger and put a conditional breakpoint in your error handler that triggers if the error is empty like that. Then you can trace back from there why that happened.

I can't think of anything off the top of my head...

[Chad Engler]

The only places in the code I see that this might be able to happen would be on a generic socket error from node:

https://github.com/englercj/node-esl/blob/master/lib/esl/Connection.js#L78
https://github.com/englercj/node-esl/blob/master/lib/esl/Connection.js#L97

Or maybe parsing XML body?

https://github.com/englercj/node-esl/blob/master/lib/esl/Parser.js#L136

Maybe the API changed for that module recently or something and I'm using it wrong now? idk...

[Chad Engler]

Closing since I haven't heard anything in a while, let me know if I need to reopen or if you solved the issue.