ava@v5.3.1 not working on Node v21.x

Question

ava@v5.3.1 not working on Node v21.x

alfonsograziano opened this issue 8 months ago · comments

Hi @sindresorhus, @novemberborn I noticed you're listed as maintainer for ava so I'd like to bring to your attention that ava@v5.3.1 tests have been failing across different systems on v21.x. Feel free to loop in more contributors that you think may help.

Here a failed run for reference:
https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/3318/testReport/

As you can see on this run jose is failing on two different nodes, but at the same time I see that a lot of nodes are skipped (["14", "ppc", "win32", "x86", "rhel", "aix", "ia32"]), so probably the problem can have a bigger scope. Can you please help us to debug the issue? I'm not sure if this is related to the new Node version or something inside the package.

In the last days we had timeout issues which has been fixed by #1003, but now this new issue looks unrelated to the timeout

Mark Wubben · Answer 1 · Thu Oct 12 2023 20:10:54 GMT+0800 (China Standard Time)

@alfonsograziano the AVA tests that are failing have to do with deterministic snapshot generation. It seems in the commit-under-test, the snapshots are 8 bytes larger. This is unexpected. I'm not sure how to run AVA against this specific commit to figure out why though.

Scanning through the Jose failures it also looks like some binary mismatch. So this could be a genuine issue with this Node.js build.

Alfonso Graziano · Answer 2 · Thu Oct 12 2023 20:39:32 GMT+0800 (China Standard Time)

Based on the Jose issue (#1011) the maintainer changed the package to make some tests non-deterministic. I'm not sure if this is the case.

You can get some other insights on Jose here: nodejs/node#50138

Can you please try to build Node.js locally and verify if you're able to reproduce the issue? Would be really helpful to understand if this is a problem with the package or a bug in the Node.js build

Simen Bekkhus · Answer 3 · Thu Oct 12 2023 20:48:21 GMT+0800 (China Standard Time)

Installing the nightly is probably easier than building from source, FWIW 🙂

Mark Wubben · Answer 4 · Fri Oct 13 2023 04:14:13 GMT+0800 (China Standard Time)

I'll need some pointers on how to access the nightly, ideally using Volta.

Simen Bekkhus · Answer 5 · Fri Oct 13 2023 14:51:32 GMT+0800 (China Standard Time)

Never heard of volta, but: https://nodejs.org/download/nightly/

Mark Wubben · Answer 6 · Fri Oct 13 2023 23:33:41 GMT+0800 (China Standard Time)

The citgm report is now under security embargo.

I ran AVA's tests using https://nodejs.org/download/nightly/v21.0.0-nightly20231009387e2929fe/ on macOS and they passed.

One thing that comes to mind is that the binary data that AVA's tests are failing for is gzip compressed so perhaps there's a change there?

Mark Wubben · Answer 7 · Mon Oct 23 2023 04:16:41 GMT+0800 (China Standard Time)

This is the same issue as discussed in nodejs/node#50138. It'll be fixed in AVA 6 but tests will fail until that is out, and I cannot provide a timeline on that.