Shrinking of large arrays uses a very large amount of memory
FeldrinH opened this issue · comments
I was trying to test a property with the following arbitrary as input:
Arbitraries.randomValue(generateValue).array(long[].class).ofMinSize(1).ofMaxSize(10_000)
// (generateValue is a custom generation function that is not relevant to the problem)The test works fine on my desktop computer, however, when running tests in a test server with only 1 GB of memory available to Java I got an OutOfMemoryError. Some memory usage profiling seems to indicate that the root cause is the use of Combinatorics.distinctPairs inside ShrinkingCommons.shrinkPairsOfElements. Oddly enough shrinkPairsOfElements is called even though values returned by Arbitraries.randomValue are unshrinkable.
Is there something that can be done to reduce the memory usage during shrinking of large arrays?
Can I somehow explicitly disable shrinking pairs of elements to limit memory usage?
PS: In this particular case adding more memory is not an option. The tests must pass with 1 GB of available memory because the ability to operate with limited memory is also a requirement being tested.
I propose that a fairly simple solution would be to generate the pairs dynamically without storing them in a list.
For example something like this:
public static Stream<Tuple2<Integer, Integer>> distinctPairs(int maxExclusive) {
if (maxExclusive < 2) {
return Stream.empty();
}
return StreamSupport.stream(new PairSpliterator(maxExclusive), false);
}
private static class PairSpliterator implements Spliterator<Tuple2<Integer, Integer>> {
private final int maxExclusive;
private int i = 0;
private int j = 1;
public PairSpliterator(int maxExclusive) {
this.maxExclusive = maxExclusive;
}
@Override
public boolean tryAdvance(Consumer<? super Tuple2<Integer, Integer>> action) {
if (j >= maxExclusive) {
return false;
}
action.accept(Tuple.of(i, j));
j += 1;
if (j >= maxExclusive) {
i += 1;
j = i + 1;
}
return true;
}
@Override
public Spliterator<Tuple2<Integer, Integer>> trySplit() {
return null;
}
@Override
public long estimateSize() {
return (long) maxExclusive * (maxExclusive - 1) / 2;
}
@Override
public int characteristics() {
return Spliterator.DISTINCT | Spliterator.ORDERED | Spliterator.SIZED | Spliterator.NONNULL | Spliterator.IMMUTABLE;
}
}
@FeldrinH Many thanks for diving into the problem and the suggestion. Since I'm on holidays I don't have the time to try it out. Maybe you want to go for a PR with the suggested change? I'd personally like a test targeting PairSplitterator to make sure it works as expected.
I'll try to look into it. I've already patched jqwik locally and know that the fixes work (or at least don't visibly break the system).
Available in "1.8.2-SNAPSHOT".
@FeldrinH Your appreciation is here: https://jqwik.net/release-notes.html#182-snapshot
Sorry about the delayed reply. I've been busy.
I have found no issues with the implementation posted here, so the PR would have been just that with a test.
Thanks for accepting this fix. Now I can get rid of the custom patches I've been using.
Released in 1.8.2
I just noticed that the release notes have a small mistake. When noting that this issue was fixed, the release notes link to issue 527, but this is issue 525.
thanks for the catch. Fixed.