jaeger: opentracing not being set in client error handlers

Question

jaeger: opentracing not being set in client error handlers

esatterwhite opened this issue 7 years ago · comments

It looks like the caller doesn't have the opentracing object set on this in the error event handlers

Description

[2017-10-24T13:14:43.335Z] FATAL (hemera-xavier-4d2a6987498e4aada870449d20fbe8a1/1353 on xavier): Fatal error
    FatalError: Fatal error
        at Hemera._sendRequestHandler (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/index.js:1052:29)
        at self._sid.self._transport.sendRequest.resp (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/index.js:1252:22)
        at Object.callback (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats/lib/nats.js:1274:5)
        at Client.processMsg (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats/lib/nats.js:1003:11)
        at Client.processInbound (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats/lib/nats.js:931:14)
        at Socket.<anonymous> (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats/lib/nats.js:470:12)
        at emitOne (events.js:115:13)
        at Socket.emit (events.js:210:7)
        at addChunk (_stream_readable.js:264:12)
        at readableAddChunk (_stream_readable.js:251:11)
    Cause: TypeError: Cannot read property 'finish' of undefined
        at Hemera.<anonymous> (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/hemera-jaeger/index.js:139:21)
        at emitOne (events.js:115:13)
        at Hemera.emit (events.js:210:7)
        at onClientPostRequest (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/extensions.js:126:11)
        at Hemera.(anonymous function).push (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/extension.js:55:15)
        at makeCall (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/fastseries/series.js:113:10)
        at ResultsHolder.release (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/fastseries/series.js:96:9)
        at Hemera.series [as _series] (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/fastseries/series.js:39:14)
        at Hemera._sendRequestHandler (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/index.js:1047:12)
        at self._sid.self._transport.sendRequest.resp (/home/esatterwhite/dev/js/node-allstar/allstar-auth/node_modules/nats-hemera/lib/index.js:1252:22)

Steps to Reproduce

Example:

Start NATS
Use plugin hemera-jeager
Send message
client handler (async function) throws error

the handler looks like this

hemera.add({
  topic: 'user'
, cmd: 'create'
, version: 'v1'
, auth$: {
    permissions: 'auth:user:create'
  }
}, async function getUser(req) {
  const opts = req
  await _validateRoles(opts.roles) // <---- This function MAY throw
  opts.roles = opts.roles || []
  if (opts.password != null) {
    this.log.debug('generating initial password for user', opts.email)
    const salt = await bcrypt.salt()
    opts.password = await bcrypt.encode(opts.password, salt)
  }
  opts.superuser = undefined
  opts.permissions = undefined
  const u = await new User(opts).save()
  const permissions = await u.getAllPermissions()
  this.log.info('new user created', u.email)

  return Object.assign({}, u, {
    password: undefined
  , permissions: permissions
  })
})

with out the jaeger pluging, it works just fine, the error is propagated back the caller. This also works fine with the jaeger plugin and there is no error.

Expected Result

error is logged and sent to jaeger

Actual Result

error is throw and process crashes

Context

Your Environment

NATS: 0.7.20
Hemera:2.2.0
NodeJs: 8.7.0
Environment name and version: ??
Operating System and version: Majaro (Arch) linux v17

Dustin Deus · Answer 1 · Wed Oct 25 2017 01:49:55 GMT+0800 (China Standard Time)

Hi, thanks for reporting I will investigate.

Dustin Deus · Answer 2 · Wed Oct 25 2017 03:11:38 GMT+0800 (China Standard Time)

@esatterwhite I can't reproduce it. I tried:

  hemera.add(
    {
      topic: 'profile',
      cmd: 'get'
    },
    async function(req) {
      throw new Error('test')
    }
  )

with this example https://github.com/hemerajs/hemera/blob/master/examples/monitoring/jaeger.js but the error is returned correctly. Please have in mind that errors that are thrown asynchronously are not able to catch.

Eric Satterwhite · Answer 3 · Wed Oct 25 2017 03:37:11 GMT+0800 (China Standard Time)

ah, sorry I mis-spoke. The function doesn't throw, it calls the reject function from a promise.
so more like

  hemera.add(
    {
      topic: 'profile',
      cmd: 'get'
    },
    async function(req) {
      await someOp()
    }
  )

function someOp() {
  return new Promise((resolve, reject) => {
    const error = new Error('broke')
    setImmediate(reject, err)
  })
}

The problem isn't that the error isn't being handled. It is getting to this line and this.opentracing is undefined

Dustin Deus · Answer 4 · Wed Oct 25 2017 04:37:06 GMT+0800 (China Standard Time)

I can't reproduce it the error is logged and the process survived. Could you provide a full example?

const Hemera = require('hemera')
const nats = require('nats').connect()
const hemeraJaeger = require('hemera-jaeger')

const hemera = new Hemera(nats, {
  logLevel: 'debug',
  childLogger: true,
  tag: 'math'
})

hemera.use(hemeraJaeger, {
  serviceName: 'math'
})
hemera.ready(() => {
  hemera.add(
    {
      topic: 'math',
      cmd: 'add'
    },
    async function(req) {
      await someOp()
    }
  )

  function someOp() {
    return new Promise((resolve, reject) => {
      const error = new Error('broke')
      setImmediate(reject, error)
    })
  }

  hemera.act({
    topic: 'math',
    cmd: 'add'
  }, function() {

  })
})

Dustin Deus · Answer 5 · Wed Oct 25 2017 04:40:24 GMT+0800 (China Standard Time)

Please also update to hemera 2.3.1

Eric Satterwhite · Answer 6 · Wed Oct 25 2017 05:54:56 GMT+0800 (China Standard Time)

Ah, I found it!. I was running a script and calling hemera.act with out wrapping it in the ready callback. I think that fixes my problem.

user error 🥇

Eric Satterwhite · Answer 7 · Wed Oct 25 2017 06:00:46 GMT+0800 (China Standard Time)

will the ready callback be executed immediately if hemera is ready ?
basically is it ok to call hemera.ready more than once or if the ready event has already been triggered.

or do you only have 1 shot?

silly example, but this:

hemera.ready(() => {
  hemera.ready(() => {
     // will I ever get here?
  })
})

Dustin Deus · Answer 8 · Wed Oct 25 2017 06:15:30 GMT+0800 (China Standard Time)

Yes, the event is called once when nats and all plugins were initialized.

Dustin Deus · Answer 9 · Wed Oct 25 2017 06:17:09 GMT+0800 (China Standard Time)

You can't listen to the ready event multiple times.

Dustin Deus · Answer 10 · Thu Oct 26 2017 23:01:09 GMT+0800 (China Standard Time)

Can be closed ?

Eric Satterwhite · Answer 11 · Thu Oct 26 2017 23:05:02 GMT+0800 (China Standard Time)

Yep. Thanks for your help