NullPointerException on ANotB

[Mikhail Lavrinovich]

IntegerSketch x = new IntegerSketch(12, 2, 0.1f, IntegerSummary.Mode.Min);
IntegerSketch y = new IntegerSketch(12, 2, 0.1f, IntegerSummary.Mode.Min);
x.update(1L, 1);
IntegerSummarySetOperations setOperations = new IntegerSummarySetOperations(IntegerSummary.Mode.Min, IntegerSummary.Mode.Min);
Intersection<IntegerSummary> intersection = new Intersection<>(setOperations);
CompactSketch<IntegerSummary> intersect = intersection.intersect(x, y);
AnotB.aNotB(x, intersect); // NPE here

java.lang.NullPointerException
	at org.apache.datasketches.HashOperations.hashArrayInsert(HashOperations.java:170)
	at org.apache.datasketches.HashOperations.convertToHashTable(HashOperations.java:341)
	at org.apache.datasketches.tuple.AnotB.getResultArraysTuple(AnotB.java:357)
	at org.apache.datasketches.tuple.AnotB.aNotB(AnotB.java:255)

[Jon Malkin]

Thanks for the report and example code, @mix74rus

Can you let us know what release version this was (or if it was current main branch)?

[Mikhail Lavrinovich]

To my mind looks like the root cause of such behaviour is that intersect sketch has null in hashArr_ (and summaryArr_ for that matter), but at the same time empty_ flag is set to false.

Going farther, such state of sketch might occur because samplingProbability was not equal to 1. (notice intentionally passed 0.1f as samplingProbability in IntegerSketch constructor) and there was no retained entries, so this code does his thing and peacefully sets empty flag to false. But i'm not quite sure about this last part.

[Mikhail Lavrinovich]

Can you let us know what release version this was (or if it was current main branch)?

Currently i'm using 2.0.0 release version and example was tested with version too. But looks like this would also reproduce in master.

[Lee Rhodes]

Mikhail, Thank you for reporting this. I can reproduce the problem, so thank you for your simple code that reproduces the issue! I am digging into the root cause and will let you know when I have it figured out. Lee.

…

On Tue, Oct 12, 2021 at 1:28 AM Mikhail Lavrinovich < ***@***.***> wrote: Can you let us know what release version this was (or if it was current main branch)? Currently i'm using 2.0.0 release version and example was tested with version too. But looks like this would also reproduce in master. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#368 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCXRQXP72BNNPLB7EGY5S3UGPWTVANCNFSM5FZ6GVVA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[Mikhail Lavrinovich]

Also I noticed that switching arguments in the last line of my example also leads to NPE but in other place.

Caused by: java.lang.NullPointerException
	at org.apache.datasketches.tuple.AnotB.getDataArraysA(AnotB.java:332) ~[datasketches-java-2.0.0.jar:?]
	at org.apache.datasketches.tuple.AnotB.aNotB(AnotB.java:248) ~[datasketches-java-2.0.0.jar:?]

Is your PR fixes that?

[Lee Rhodes]

Your issue inspired me to look deeper where I have found a number of very subtle corner-case issues, which I am working to resolve. I am pretty confident I have uncovered them all. However, at the same time I am taking the time to do some refactoring that will make it easier to test and understand the code in the future. Please be patient :)

[Lee Rhodes]

This bug should be considered fixed and is currently in master awaiting the next release.