array index error on ReqSketch
jhsenjaliya opened this issue · comments
ReqSketch often throws error on small dataset.
only thing I explicitly set is setHighRankAccuracy(true)
anyone facing this issues? its occasional so hard to understand any pattern, will try to create a test.
java.lang.NegativeArraySizeException: -328
at org.apache.datasketches.req.ReqCompactor.toByteArray(ReqCompactor.java:295)
at org.apache.datasketches.req.ReqSerDe.toByteArray(ReqSerDe.java:305)
at org.apache.datasketches.req.ReqSketch.toByteArray(ReqSketch.java:418)
Do you have an example of a dataset that triggers this behavior for you that we can use for validation?
unfortunately I was not able to reproduce with the same dataset on single process,
this is while serialization, so let me gather more info with some debug logs. Thanks for chiming in though
closing the issue for now. Looks like its related to the framework that is using this class whiile serialization.
This seems to be random actually, I thought this is due to multi threaded distributed environment where serialization is performed, but I am seeing this even in a single thread and single node. and can not figure out why the negative index would ever happen. The error comes from "ReqCompactor.java:295" on ( version 5.0.2 ) where bytes are negative that are calculated via function getSerializationBytes()
-- still trying to figure out how to debug this. would really appreciate any help. Thanks
Please help us reproduce this by providing a simple example with data. --Thanks