pta reports PASS 0 before passing Zora tests finish running

[AJ ONeal]

I'm a little confused because this outputs PASS: 0 before Zora finishes running the very first test (which passes with 2 tests).

What did I do wrong?

> dashtx@0.10.3 test
> npx -p pta@1.x -- pta 'tests/**/*.js'


  TOTAL:  0
   PASS:  0
   FAIL:  0
   SKIP:  0

TAP version 13
# inputs sort correctly
ok 1 - inputs are sorted as expected
# outputs sort correctly
ok 2 - outputs are sorted as expected

1..2
# tests 2
# pass  2
# fail  0
# skip  0

[AJ ONeal]

I meant to post this as a pta issue.

[RENARD Laurent]

Thanks for the report. Do you have the same problem if you install pta on your machine and run it directrly (without npx) ?

[AJ ONeal]

There's no difference between the two:
makes no difference whether I run npx -p pta@1 -- pta './tests/**/*.js' or pta './tests/**/*.js'.

Versions

node --version
v20.4.0

uname -srm
Darwin 22.5.0 arm64

"zora": "^5.2.0"

pta --version
(error, but it's 1.2 or so)

Install pta

# to prevent from mistakenly polluting the `package.json`:
pushd /tmp/
npm install --location=global pta@1
popd

Install Zora

npm install --save-dev zora@5

Bogo tests

./tests/test.js:

"use strict";

let Zora = require("zora");

Zora.test("sync:pass", function (t) {
  t.ok(true);
});

Zora.test("sync:fail", function (t) {
  t.ok(false);
});

Zora.test("sync:timeout", function () {
  // do nothing
});

Zora.test("async:pass", async function (t) {
  await sleep(50);
  t.ok(true);
});

Zora.test("async:fail", async function (t) {
  await sleep(50);
  t.ok(false);
});

Zora.test("async:timeout", async function () {
  await sleep(50);
  // do nothing
});

Zora.test("cb:pass", function (t) {
  let p = sleep(50).then(function () {
    t.ok(true);
  });
  return p;
});

Zora.test("cb:fail", function (t) {
  let p = sleep(50).then(function () {
    t.ok(false);
  });
  return p;
});

Zora.test("cb:timeout", function () {
  let p = sleep(50).then(function () {
    // do nothing
  });
  return p;
});

async function sleep(ms) {
  return await new Promise(function (resolve) {
    setTimeout(resolve, ms);
  });
}

Run tests

npm ci
pta 'tests/**/*.js'

Result

  TOTAL:  0
   PASS:  0
   FAIL:  0
   SKIP:  0

TAP version 13
# sync:pass
ok 1 - should be truthy
# sync:fail
not ok 2 - should be truthy
  ---
    actual: false
    expected: "truthy value"
    operator: "ok"
    at: " /private/tmp/base58check.js/tests/test.js:10:5"
  ...
# sync:timeout
# async:pass
ok 3 - should be truthy
# async:fail
not ok 4 - should be truthy
  ---
    actual: false
    expected: "truthy value"
    operator: "ok"
    at: " /private/tmp/base58check.js/tests/test.js:24:5"
  ...
# async:timeout
# cb:pass
ok 5 - should be truthy
# cb:fail
not ok 6 - should be truthy
  ---
    actual: false
    expected: "truthy value"
    operator: "ok"
    at: " /private/tmp/base58check.js/tests/test.js:41:7"
  ...
# cb:timeout

1..6
# tests 6
# pass  3
# fail  3
# skip  0

[AJ ONeal]

Also note that zora sums the number of tests incorrectly:

# tests 6
# pass  3
# fail  3
# skip  0

There were 9 tests. 3 should have passed and 6 should have failed (3 timeouts).

[RENARD Laurent]

Also note that zora sums the number of tests incorrectly:

# tests 6
# pass  3
# fail  3
# skip  0

There were 9 tests. 3 should have passed and 6 should have failed (3 timeouts).

Note quite. In TAP protocol, a "test" is actually an assertion. Your timeout cases have no assertion.

There is indeed a timeout check but it is by default 5s not 50ms. You can overwrite that value by test, or globally with an env variable (or with a command line option when using pta). see documentation

[RENARD Laurent]

There's no difference between the two:
makes no difference whether I run npx -p pta@1 -- pta './tests//*.js' or pta './tests//*.js'.

I see the problem. Pta imports zora singleton to hold its reporting and load the spec files. With your setups it loads zora from a different file and therefore "hold" a different singleton (because we need a file for commonjs, es, etc..).

You can either install pta locally to your project or as you use commonjs, get the same result with basic bash command in your npm script:

ZORA_REPORTER=json node -r **/*.test.js | zr

where "zr" is zora reporter you can install npm install -g zora-reporters