Giters

xerial / snappy-java

Snappy compressor/decompressor for Java

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

pure java compressed output cannot be decompressed by native

bokken opened this issue · comments

commented

Content compressed with the pure java implementation fails to decompress by the native code with error
FAILED_TO_UNCOMPRESS(5).

Here is a unit test which causes the failure. It is the testSnappyCompressNativeDecompress which fails.

package org.xerial.snappy;

import static org.junit.Assert.assertEquals;

import java.nio.ByteBuffer;
import java.util.Random;

import org.junit.Test;
import org.xerial.snappy.pure.PureJavaSnappy;

public class PureSnappyCompatTest
{
    private static final byte[] DATA = new byte[34873];

    static
    {
        int i = 0;
        final Random random = new Random(87);
        for ( ; i<DATA.length - 6; i+=7)
        {
            if (i % 8 == 1)
            {
                DATA[i] = (byte) random.nextInt(256);
                DATA[i + 1] = (byte) random.nextInt(256);
                DATA[i + 2] = (byte) random.nextInt(256);
                DATA[i + 3] = (byte) random.nextInt(256);
                DATA[i + 4] = (byte) random.nextInt(256);
                DATA[i + 5] = (byte) random.nextInt(256);
                DATA[i + 6] = (byte) random.nextInt(256);
            }
            else
            {
                DATA[i] = 1;
                DATA[i + 1] = 1;
                DATA[i + 2] = 1;
                DATA[i + 3] = 2;
                DATA[i + 4] = 2;
                DATA[i + 5] = 2;
                DATA[i + 6] = 2;
            }
        }
        while(i<DATA.length)
        {
            DATA[i++] = (byte) random.nextInt(256);
        }
    }
    
    private final SnappyApi snappyNative = SnappyLoader.loadSnappyApi();
    private final SnappyApi snappyJava = new PureJavaSnappy();

    @Test
    public void testNativeCompressJavaDecompress() throws Exception
    {
        ByteBuffer raw = ByteBuffer.allocateDirect(DATA.length);
        ByteBuffer compressed = ByteBuffer.allocateDirect(Snappy.maxCompressedLength(DATA.length));
        
        raw.put(DATA);
        raw.flip();

        final int length = snappyNative.rawCompress(raw, 0, raw.remaining(), compressed, 0);
        compressed.limit(length);

        ByteBuffer actual = ByteBuffer.allocateDirect(DATA.length);
        snappyJava.rawUncompress(compressed, 0, compressed.remaining(), actual, 0);

        actual.flip();
        assertRawData(0, DATA.length, actual);
    }

    @Test
    public void testSnappyCompressNativeDecompress() throws Exception
    {
        ByteBuffer raw = ByteBuffer.allocateDirect(DATA.length);
        ByteBuffer compressed = ByteBuffer.allocateDirect(Snappy.maxCompressedLength(DATA.length));
        
        raw.put(DATA);
        raw.flip();

        final int length = snappyJava.rawCompress(raw, 0, raw.remaining(), compressed, 0);
        compressed.limit(length);

        ByteBuffer actual = ByteBuffer.allocateDirect(DATA.length);
        snappyNative.rawUncompress(compressed, 0, compressed.remaining(), actual, 0);

        actual.flip();
        assertRawData(0, DATA.length, actual);
    }

    private static void assertRawData(int offset, int length, ByteBuffer actual)
    {
        assertEquals(length, actual.remaining());
        for (int i = 0; i<length; ++i)
        {
            assertEquals("value at idx: " + i, DATA[i + offset], actual.get());
        }
    }
}
commented

@xerial, this pure java implementation has been problematic.
I am not sure it is a great idea to fall back to using it by default. At a minimum, I would like to have a programmatic way to tell if that is what loaded (so that i can avoid using it).

commented

The existing PureJava tests do not /actually/ test the pure java implementation. This is because the PureJavaSnappyTestUtil.setUp only gets the new impl into SnappyLoader, not into the static value in Snappy.

Changing setUp to call Snappy.init() instead (like what is done in tearDown) will cause the PureJava implementation to be used. This causes lots of test failures.

commented

Thanks for reporting. That's totally unexpected... We may need to drop pure-java support ....

commented

@xerial, my vote (for what it is worth), is a really quick release with pure java stuff removed.
Then look at bringing back parts/pieces as they can be proven to work reliably.

commented

@xerial can we get something done soon on this?

commented

@xerial this continues to cause problems. Can we get the pure java implementation removed (either directly or via code)?

commented

@xerial Any updates on this? We also had a better feeling if the pure java implementation were removed or at least that we could disable the pure java implementation fallback.

commented

It seems forking the library is best bet here...