BitShuffle#shuffle for arrays of primitive datatypes does not consider the byte-order (leading to wrong results)
foosmate opened this issue · comments
The underlying BitShuffle library works internally with an algorithm that was designed for the little-endian format. For this reason, the input data must always be passed in little-endian format.
However, the snappy-java wrapper directly passes an int[] array with the native byte order. Thus the Java implementation does not give the same results on little-endian and big-endian platforms for the shuffle / unshuffle methods.
The shuffle/unshuffle methods still work bijective so this issue is not easily noticeable.
How to fix:
The data array based on primitive types (e.g. int[], short[], long[], etc.) must be converted to little-endian format in the Java implementation if executed on big-endian platforms (before passing the buffer to the external library).
Related #443