`test_compression` fails on the latest Fedora Rawhide

[Frantisek Sumsal]

Hey!

A while ago I got a notification that the botan2 package fails to build on Fedora Rawhide due to failing make check, and while trying to finally address it I noticed that it is reproducible with the latest master branch as well:

$ podman run -it --rm fedora:rawhide
# dnf install -y git dnf-plugins-core
# dnf builddep -y botan2
# git clone https://github.com/randombit/botan --depth=1
# cd botan
# ./configure.py --enable-modules=bzip2,zlib
# make -j $(nproc)
# make check |& tee log.txt
...
ZFEC encoding/decoding ran 10449 tests in 6.43 msec all ok
Tests complete ran 2938384 tests in 22.04 sec 3 tests failed (in compression_tests)
ERROR:root:Error running ./botan-test --data-dir=./src/tests/data
make: *** [Makefile:65: check] Error 1
# grep "Failure" log.txt
Failure 1: deflate compression Level 9 compresses text at least as well as level 1 unexpected result 443 < 444 (at src/tests/test_compression.cpp:164)
Failure 1: gzip compression Level 9 compresses text at least as well as level 1 unexpected result 461 < 462 (at src/tests/test_compression.cpp:164)
Failure 1: zlib compression Level 9 compresses text at least as well as level 1 unexpected result 449 < 450 (at src/tests/test_compression.cpp:164)

[René Meusel]

Admittedly, that test seems quite opinionated on how a compression library should work and how its performance characteristics should look like. Botan just wraps the compression libraries and the test should probably just make sure that this wrapper works as expected.

From what I can tell, we could just remove the "opinionated" portion of this test. But perhaps, @randombit could add some context on that?

[Jack Lloyd]

I guess this is opinionated but I am surprised that level 1 compression compresses BETTER THAN level 9 compression. This kind of implies we should just use level 1 and save the extra CPU costs ....

[Frantisek Sumsal]

I guess this is opinionated but I am surprised that level 1 compression compresses BETTER THAN level 9 compression. This kind of implies we should just use level 1 and save the extra CPU costs ....

Yeah, I find this really suspicious, and I still feel like there's a bug somewhere else. But it's weird that it fails the same way with three different compression methods.

[Frantisek Sumsal]

After some digging I'd say that the culprit here is the transition from zlib to zlib-ng in current Fedora Rawhide (I just didn't realize that zlib, gzip, and deflate are all provided by zlib until now). There is a zlib-ng-compat package, which is supposed to provide a zlib-compatible API and ABI, but there's clearly a bug somewhere. I'll try to put together some reproducer without botan to see if that's really the case.

[Frantisek Sumsal]

Yup, it seems to be a bug in zlib-ng:

#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <zlib.h>

const char* const COMPRESSION_TEST_TEXT =
   "'Twas brillig, and the slithy toves"
   "Did gyre and gimble in the wabe:"
   "All mimsy were the borogoves,"
   "And the mome raths outgrabe."

   "'Beware the Jabberwock, my son!"
   "The jaws that bite, the claws that catch!"
   "Beware the Jubjub bird, and shun"
   "The frumious Bandersnatch!'"

   "He took his vorpal sword in hand;"
   "Long time the manxome foe he sought-"
   "So rested he by the Tumtum tree"
   "And stood awhile in thought."

   "And, as in uffish thought he stood,"
   "The Jabberwock, with eyes of flame,"
   "Came whiffling through the tulgey wood,"
   "And burbled as it came!"

   "One, two! One, two! And through and through"
   "The vorpal blade went snicker-snack!"
   "He left it dead, and with its head"
   "He went galumphing back."

   "'And hast thou slain the Jabberwock?"
   "Come to my arms, my beamish boy!"
   "O frabjous day! Callooh! Callay!'"
   "He chortled in his joy."

   "'Twas brillig, and the slithy toves"
   "Did gyre and gimble in the wabe:"
   "All mimsy were the borogoves,"
   "And the mome raths outgrabe.";


int main(void) {
    Byte *out_l1, *out_l9;
    uLong out_l1_len, out_l9_len;

    out_l1_len = out_l9_len = compressBound(strlen(COMPRESSION_TEST_TEXT));
    assert(out_l1 = calloc(out_l1_len, 1));
    assert(out_l9 = calloc(out_l9_len, 1));

    assert(compress2(out_l1, &out_l1_len, COMPRESSION_TEST_TEXT, strlen(COMPRESSION_TEST_TEXT), 1) == Z_OK);
    assert(compress2(out_l9, &out_l9_len, COMPRESSION_TEST_TEXT, strlen(COMPRESSION_TEST_TEXT), 9) == Z_OK);
    printf("Input: %lu bytes\nL1 output: %lu bytes\nL9 output: %lu bytes\n",
           strlen(COMPRESSION_TEST_TEXT), out_l1_len, out_l9_len);

    assert(out_l1_len > out_l9_len);

    return 0;
}

On F38 (with zlib-1.2.13-4.fc39.x86_64):

$ gcc -o test -lz test.c
$ ./test 
Input: 900 bytes
L1 output: 455 bytes
L9 output: 450 bytes

On Fedora Rawhide (with zlib-ng-compat-2.1.6-1.fc40.x86_64):

# gcc -o test -lz test.c
# ./test 
Input: 900 bytes
L1 output: 449 bytes
L9 output: 450 bytes
test: test.c:57: main: Assertion `out_l1_len > out_l9_len' failed.
Aborted (core dumped)

So I'll go ahead, close this issue, and report this to zlib-ng instead.

[Frantisek Sumsal]

Reopening again, as the behavior is not actually a bug, but an implementation detail, please see the full response in the zlib-ng ticket: zlib-ng/zlib-ng#1656 (comment).

[René Meusel]

Wow! Thanks for digging down this rabbit hole, both @mrc0mmand and @Dead2. From what I can tell, that's really just Hyrum's Law in practice (xkcd 1172) and there's no real reason to keep this test so specific. @randombit I'll cook up a pull request to relax it.

@mrc0mmand For now, as a temporary workaround, you may get away with running ./botan-test with --skip-tests=compression_tests. That should disable the failing test.