rollbar / rollbar.js

Hello,

We're using rollbar.js in a number of AWS lambdas. We will sometimes get the following error:

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [ClientRequest]. Use emitter.setMaxListeners() to increase limit
(Use `node --trace-warnings ...` to show where the warning was created)

This error appears to occur after sending 5+ (the exact number varies) errors within a few minutes of each other, using the following:

rollbar.log(filteredError, customFields, (err, data) => {
  if (err) {
    reject(err)
  } else {
    resolve(data)
  }
})

From what I can gather from the code here, it's being thrown because of this line:

rollbar.js/src/server/telemetry.js

Line 100 in 759af08

req.on('error', err => {

This line was introduced in this PR: #943

I don't know why var req = orig.apply(https, wrappedArgs); (2 lines up) is returning the same ClientRequest object each time - but it must be, because that's why this error is occurring. You can duplicate something similar locally with the following:

var req = http.request('http://www.google.com')
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})
req.on('error', () => {})

I can try to create a small example repo is this isn't sufficient.

Thanks for the detailed report and repro. This should be enough detail.

@zjullion I think I see what's happening.

The code path where the listener is added wraps all calls to http.request() and adds the listener to detect and report any errors.

If the wrapper is initialized twice (or more), you will get nested wrappers. Each nested wrapper will add it's own listener, and all of these listeners are added to the same request.

Normally this would never happen, but I think with Lambda warm start, Rollbar's init code runs on each start but the previous http.request() wrappers are never removed. Hence the nested wrappers.

If your Lambda function is structured so that you can run code at exit, this would be a short term workaround to run on exit:
rollbar.instrumenter.deinstrumentNetwork && rollbar.instrumenter.deinstrumentNetwork()

I'll look into a better long term fix, and link here when there's a PR.

Thanks for the update! This is just a background error for us now, so we aren't in a huge hurry to make any changes, but we will update our rollbar version when a fix becomes available.

I had a similar issue when running my tests (using Jest) in CI (Github Actions). Some of my tests were integration tests and I was testing failure cases, which happened to have rollbar involved. After realizing that the EventEmitter warnings were being caused by roller I just turned off autoInstrument config when running in test environment, and that seems to have solved the problem for me. I will investigate further later on if my testing approach should be revisited completely.

MaxListenersExceededWarning being thrown by src/server/telemetry.js