`--jobs` has extra rebuilds of dependencies, causing build timeouts

[Kevin Reid]

Today I tried using cargo mutants -j N, and noticed that it caused the build timeout to be hit. The timeouts were not just due to additional system load, but because N-1 of the build jobs started rebuilding from scratch. Look at this example output from mid-run:

$ cargo mutants -p all-is-cubes-base -f all-is-cubes-base/src/math/octant.rs -j4
Found 96 mutants to test
ok       Unmutated baseline in 14.4s build + 9.5s test
 INFO Auto-set build timeout to 29s
 INFO Auto-set test timeout to 48s
build    all-is-cubes-base/src/math/octant.rs:232:38: replace & with ^ in OctantMask::collapse_to_positive ... 1.8s
└           Compiling all-is-cubes-base v0.7.1 (/var/folders/bm/qff3m4w11hj5p03sytlsm8nw0000gn/T/cargo-mutants-all-is-cubes-mlNFhN.tmp/all-
build    all-is-cubes-base/src/math/octant.rs:73:13: replace | with ^ in Octant::of_vector ... 1.8s
└           Compiling indoc v2.0.5
build    all-is-cubes-base/src/math/octant.rs:173:51: replace >> with << in OctantMask::shift ... 1.8s
└           Compiling indoc v2.0.5
build    all-is-cubes-base/src/math/octant.rs:93:39: replace == with != in Octant::reflect ... 1.8s
└           Compiling diff v0.1.13
0/96 mutants tested, 26s elapsed

Notice that the first job is rebuilding all-is-cubes-base, but the other three are compiling its dependencies from crates.io. (Presumably this is because each job has its own target directory?) This means those builds are longer, and (in my case) likely to hit the build timeout.

Perhaps the build timeout should be extended for those initial jobs which have fresh target directories.

[Martin Pool]

Hi,

Yes, unfortunately it seems like cargo will always rebuild everything when the directory is copied or even renamed.

And yes, we should increase the auto-set build timeout.

[Kevin Reid]

Some alternative ideas that might be more reliable than purely extending the timeout:

• Run a baseline build in each target directory before starting any mutation (this adds to the total time, but hopefully not very much)
• Don't start measuring the start point until Cargo indicates it's compiling the crate that was mutated (but this requires parsing the output)

[Martin Pool]

I started on making it run a baseline build in each directory. This might be a good approach, but it might also add significantly to the time: since each cargo run generally can generate quite a lot of concurrent jobs we typically spend n_jobs * clean_build time, without benefiting much from running them in parallel.

Beyond this, perhaps we could take a broader look at how to detect compile-time hangs caused by mutations. In a sense this is all a workaround for disabling the lint that limits time spent processing constant functions.