Callback called multiple times in .exec

[Akseli Palén]

We use command-line-test in genversion to test genversion's CLI. Today I extended our test suite with some negative test cases and found unexpected behavior on command-line-test's part. After a brief inspection it seems to me that due to promise handling it is possible for clitest.exec to call the callback twice: first time without an error and second time with an error that has happened in the first callback.

The culprit: command-line-test.js#L144

The behavior I would expect is that clitest.exec(...) calls the callback only once. Clitest should only be responsible to handle its own errors, not the errors that might happen in the callbacks. The possible errors in the callback should be handled by the test framework (mocha in my case) or not at all.

Here is a stripped-down snippet from our test suite to replicate the issue. The test suite has a bug which throws an error.

it('should work', done => {
  const clitest = new CliTest()
  clitest.exec('genversion --some-flag some-input.js', (err, response) => {
    if (err) {
      // The second callback call goes here.
      // It looks like there was an error in running 'genversion' command but in
      // reality the error was caused by the bug in the test code below.
      console.log('bar')
      return done(err)
    }

    // The first callback has err=null and therefore execution reaches here.
    // However, there is an error in the test code, which throws an Error.
    console.log('foo')
    throw new Error('Some error in the test')
    return done()
 })

The code above produces:

foo
bar

The problem is that this situation is hard to detect and understand. We expect callbacks to be called only once.

I am not sure how this should be resolved though. One option, probably unsatisfactory, is to replace the following callback handling around command-line-test.js#L144:

promise.then(data => {
  callback.call(this, null, data);
}).catch(err => {
  callback.call(this, `exec ${command} error with: ${err}`);
});

with something like this:

promise.then(data => {
  try {
    callback.call(this, null, data);
  } catch (externalError) {
    console.error(externalError);
  }
}).catch(err => {
  callback.call(this, `exec ${command} error with: ${err}`);
});

Alternatively, something like this could work too (see https://stackoverflow.com/a/30741722/638546):

promise.then(data => {
  try {
    callback.call(this, null, data);
  } catch (externalError) {
    setTimeout(() => {
      throw externalError;
    }, 0);
  }
}).catch(err => {
  callback.call(this, `exec ${command} error with: ${err}`);
});

Any thoughts?