BUG: lz4_compresses function always fails
pete4abw opened this issue · comments
From master 0.640
Starting thread 0 to compress 128700872 bytes from stream 1
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 128700872. Compressed size = 20480.00% of chunk, 1 Passes
lz4 testing FAILED for chunk 13459191. Compressed size = 83886079.96% of chunk, 1 Passes
from 0.631 using lzo1x_compress
lzo testing OK for chunk 128700872. Compressed size = 71.18% of chunk, 1 Passes
lzo testing OK for chunk 128700872. Compressed size = 41.46% of chunk, 1 Passes
lzo testing OK for chunk 128700872. Compressed size = 39.46% of chunk, 1 Passes
lzo testing OK for chunk 128700872. Compressed size = 35.97% of chunk, 1 Passes
lzo testing OK for chunk 128700872. Compressed size = 35.52% of chunk, 1 Passes
lzo testing OK for chunk 128700872. Compressed size = 28.37% of chunk, 1 Passes
lzo testing OK for chunk 13459191. Compressed size = 72.42% of chunk, 1 Passes
The issue appears to be using test_len instead of in_len in test function. Really broken because it will fail on first attempt.
lz4_ret = LZ4_compress_default((const char *)test_buf, c_buf, test_len, dlen);
if (!lz4_ret) // Bigger than dlen, no point going further
break;
if (lz4_ret < best_dlen)
best_dlen = lz4_ret;
if (lz4_ret < test_len) {
ret = 1;
break;
}
Try this
lz4_ret = LZ4_compress_default((const char *)test_buf, c_buf, in_len, dlen);
if (!lz4_ret) // Bigger than dlen, no point going further
try again;
Breaking out before test_len is 0 is just wrong
What file are you testing? That is the correct way to use lz4 compress which has different semantics to lzo, but lz4 will be less able to compress compared to lzo.
What file are you testing? That is the correct way to use lz4 compress which has different semantics to lzo, but lz4 will be less able to compress compared to lzo.
Just a kernel source, 5.10.y from git and tar'red. I just cloned this repo and ran. I (no surprise) use a different logic, but it works with lz4 as expected. You must use in_len in the function call and remove the first failure test because if the first test fails, the function will abort. If you decide to fix Threshold testing, you will need to test all parts of the source buffer. Having the !lz4_ret test will abort the run.
Here with in_len
lz4 testing OK for chunk 128700872. Compressed size = 74.64% of chunk, 1 Passes
lz4 testing OK for chunk 128700872. Compressed size = 42.24% of chunk, 1 Passes
lz4 testing OK for chunk 128700872. Compressed size = 39.09% of chunk, 1 Passes
lz4 testing OK for chunk 128700872. Compressed size = 35.25% of chunk, 1 Passes
lz4 testing OK for chunk 128700872. Compressed size = 35.25% of chunk, 1 Passes
lz4 testing OK for chunk 128700872. Compressed size = 27.87% of chunk, 1 Passes
You're right, well spotted. Will have to churn a quick bugfix release. Thanks.
diff --git a/stream.c b/stream.c
index a2ec0f0..61e0fab 100644
--- a/stream.c
+++ b/stream.c
@@ -1907,9 +1907,9 @@ static int lz4_compresses(rzip_control *control, uchar *s_buf, i64 s_len)
int lz4_ret;
workcounter++;
- lz4_ret = LZ4_compress_default((const char *)test_buf, c_buf, test_len, dlen);
- if (!lz4_ret) // Bigger than dlen, no point going further
- break;
+ lz4_ret = LZ4_compress_default((const char *)test_buf, c_buf, in_len, dlen);
+ // if (!lz4_ret) // Bigger than dlen, no point going further
+ // break;
if (lz4_ret < best_dlen)
best_dlen = lz4_ret;
if (lz4_ret < test_len) {
Fixed in 0.641 with a reworked lz4_compresses.