Non-termination of test execution in Jest when tested function mutates its arguments

[Niklas Gruhn]

🐛 Bug Report

In some situations test execution doesn't appear to terminate. It also does not run into a timeout. I suspect this can happen when the tested function mutates its arguments. I have only observed this with Jest, so I'm not sure if this is rather a Jest issue.

To Reproduce

Consider the following example (full setup available in this repository):

const fc = require('fast-check')

// the function to be tested
function nasty(array) {
    array.push(1) // nasty argument mutation
    return array
}

test('demo test', () => {
    fc.assert(fc.property(
        fc.array(fc.integer()),
        array_before => {
            const array_after = nasty(array_before)

            // some random assertion that should eventually fail
            expect(array_after.length).toBeLessThenOrEqual(3)
        }
    ))
})

If I run this with Jest, it looks like this. Note, if I pass a copy of array_before into nasty, the test run terminates immediately (with a failure as expected). So I really think the argument mutation is the offender here.

Screen.Recording.2022-09-26.at.16.23.34.mov

Your environment

Packages / Softwares	Version(s)
fast-check	3.1.4
node	v16.0.5
jest	29.0.3

[Nicolas DUBIEN]

Hey,

Thanks a lot for the report, but this is actually something fully expected as written in the official documentation. See:

The predicate function must not change the inputs it received. If it needs to, it has to clone them before going on. Impacting the inputs might led to bad shrinking and wrong display on error.

Link: https://github.com/dubzzz/fast-check/blob/main/packages/fast-check/documentation/Runners.md#properties

Unfortunately patching this kind of issue would have an overall negative performance impact for all tests being written via fast-check. Meanwhile I could probably come with some kind of helper to at least detect poisoned inputs (as the one I already provide to detect poisoned globals via @fast-check/poisoning).

[Niklas Gruhn]

I see... Thanks for the quick response. But do you understand why it's not terminating? Shouldn't Jests timeout kick in eventually?

[Nicolas DUBIEN]

If I'm not wrong, by default, Jest is unable to kill a test running into an infinite loop so it might be the reason why the test cannot be stopped. But they might have a worker based version that can do that (not sure).

[Nicolas DUBIEN]

Regarding this issue on itself, I'll probably check to see if I can make array's shrink more resilient to external modifications. But fixing it or not will highly depends on the overall runtime cost it can add.

[Niklas Gruhn]

Makes sense. Close this if you want. Thanks you!

[Nicolas DUBIEN]

Let's keep it opened for now so that I can have a reminder that there is possibly a quick improvement to do for the shrinker of arrays.

[Nicolas DUBIEN]

Good news: I have some ideas to make it work with, I hope, no runtime footprint but probably a huge memory footprint as I'll have to keep one copy of the array in memory to make the code work. I'll work on it soon and assess impacts to check whether acceptable or not. The risk of side effects being huge, it does not seem that an issue to try to limit that kind of issues to the end user a little bit more if feasible without any excessive impact.

[Nicolas DUBIEN]

I'll close it for now and probably take it back later as part of a more important set of changes