ArrayIndexOutofBoundException while updating DataSketch
architmmmec opened this issue · comments
doublesSketch = DoublesSketch.builder().build();
private double[] doubleArray = { 0.99 };
public void updateSketch(long latency) {
doublesSketch.update(latency);
System.out.println("Latency " + latency);
}
private double getPercentile() {
double value = doublesSketch.getQuantiles(doubleArray)[0];
System.out.println("Percentile " + value);
doublesSketch.reset();
return value;
}
Quantiles HeapUpdateDoublesSketch SUMMARY:
Empty : false
Direct, Capacity bytes : false,
Estimation Mode : true
K : 128
N : 510
Levels (Needed, Total, Valid): 1, 1, 1
Level Bit Pattern : 1
BaseBufferCount : 255
Combined Buffer Capacity : 384
Retained Items : 382
Compact Storage Bytes : 3,088
Updatable Storage Bytes : 3,104
Normalized Rank Error : 1.406%
Normalized Rank Error (PMF) : 1.711%
Min Value : 3.380000e+02
Max Value : 5.729000e+03
END SKETCH SUMMARY
java.lang.ArrayIndexOutOfBoundsException: 384
at com.yahoo.sketches.quantiles.HeapDoublesSketchAccessor.set(HeapDoublesSketchAccessor.java:40)
at com.yahoo.sketches.quantiles.DoublesUpdateImpl.zipSize2KBuffer(DoublesUpdateImpl.java:122)
at com.yahoo.sketches.quantiles.DoublesUpdateImpl.inPlacePropagateCarry(DoublesUpdateImpl.java:96)
at com.yahoo.sketches.quantiles.HeapUpdateDoublesSketch.update(HeapUpdateDoublesSketch.java:222)
at MetricCache.updateSketch
@leerho @AlexanderSaydakov can you pls help here
The code is incomplete. It is not clear what sequence of operations led to the error. It would also be helpful to print the sketch before the problematic operation.
@AlexanderSaydakov , we are posting metric every minute, so updateSketch is called on every insertion to DB(having latency) and getPercentile is called every minute. I hope it answers your question. And we are resetting the sktech every minute. Are we missing anything here?
It is not clear what sequence of operations led to the error. I would also suggest to print the sketch before the problematic operation.
@AlexanderSaydakov , exception is coming on this line doublesSketch.update(latency);
Double skecth
Quantiles HeapUpdateDoublesSketch SUMMARY:
Empty : false
Direct, Capacity bytes : false,
Estimation Mode : true
K : 128
N : 510
Levels (Needed, Total, Valid): 1, 1, 1
Level Bit Pattern : 1
BaseBufferCount : 255
Combined Buffer Capacity : 384
Retained Items : 382
Compact Storage Bytes : 3,088
Updatable Storage Bytes : 3,104
Normalized Rank Error : 1.406%
Normalized Rank Error (PMF) : 1.711%
Min Value : 3.380000e+02
Max Value : 5.729000e+03
END SKETCH SUMMARY
java.lang.ArrayIndexOutOfBoundsException: 384
at com.yahoo.sketches.quantiles.HeapDoublesSketchAccessor.set(HeapDoublesSketchAccessor.java:40)
at com.yahoo.sketches.quantiles.DoublesUpdateImpl.zipSize2KBuffer(DoublesUpdateImpl.java:122)
at com.yahoo.sketches.quantiles.DoublesUpdateImpl.inPlacePropagateCarry(DoublesUpdateImpl.java:96)
at com.yahoo.sketches.quantiles.HeapUpdateDoublesSketch.update(HeapUpdateDoublesSketch.java:222)
at MetricCache.updateSketch
@AlexanderSaydakov can you pls check
Are you operating in a multi-threaded environment?
Sketches are not thread-safe.
What is the system environment you are operating in? Spark? Druid? etc.
We are using it in project reactor code. What is the alternative for multithreaded environment?
You will need to wrap the sketch in a synchronized wrapper. You only need to synchronize the methods that you actually use.
We have not heard any feedback in the last 2 days, so I will close this issue. If necessary, we can always reopen it.