Different definition of BITBUF_NBITS in deflate_compress/deflate_decompress

[Pavel P]

Would it be ok to modify one of them so these are identical for the files, or undef it at the end of the files?

vs

https://github.com/ebiggers/libdeflate/blob/master/lib/deflate_decompress.c#L110
https://github.com/ebiggers/libdeflate/blob/master/lib/deflate_compress.c#L649

[Eric Biggers]

They are in different .c files, so there should be no problem. What problem are you trying to solve?

[Pavel P]

I do amalgamated build where all .c files are included into a single file: https://github.com/pps83/libdeflate/blob/master/libdeflate.c

[Pavel P]

FYI, I just tried latest libdeflate compiled with clang-cl, mingw64 (gcc version 12.2.0), and msvc 2019.

First, the test time with older libdeflate that didn't have msvc optimiations enabled: processing time: 51308.601ms
After I updated, time for vs2019: processing time: 38454.476ms (epic 25% improvement!)

Clang-cl (with vs2022): processing time: 40167.472ms
mingw64: processing time: 40075.764ms

For each of the builds I made a few runs, so, surprisingly, ms compiler is actually ~4% faster than mingw64/clang in my case.

I tried to profile my test, libdeflate_deflate_decompress is 45% of the runtime (my test loads thousands of zip files, extracts json file from them, and computes hash of the data). So, that means that actual improvement is much bigger than 25%

[Pavel P]

I didn't try vs2022, for some reason x64 release build mysteriously fails linking for me.

[Eric Biggers]

I do amalgamated build where all .c files are included into a single file: https://github.com/pps83/libdeflate/blob/master/libdeflate.c

That is not something that is supported currently. If it were to be supported, it would need to be tested in CI.

I suppose there wouldn't be too many issues with just concatenating the .c files. However that would not be a real single file amalgamation, since it would still #include all the internal header files. A real single file amalgamation, like the one that SQLite provides, would incorporate all the internal header files into the single .c file too. That would require much more work to support.

What is your use case for a single file amalgamation?

If you are hoping for a performance boost due to function inlining, you'll almost certainly find that there isn't any measurable difference. The different parts of libdeflate are fairly separate from each other, and each function tends to be fairly big.

I tried to profile my test, libdeflate_deflate_decompress is 45% of the runtime (my test loads thousands of zip files, extracts json file from them, and computes hash of the data). So, that means that actual improvement is much bigger than 25%

Great! Though, I'm surprised that the recent MSVC support improvements would have made libdeflate_deflate_decompress that much faster, as they mainly improved the performance of checksumming. Probably you're actually using libdeflate_zlib_decompress?

I didn't try vs2022, for some reason x64 release build mysteriously fails linking for me.

If it is a problem on libdeflate's side, please report it.

[Pavel P]

I do amalgamated build where all .c files are included into a single file: https://github.com/pps83/libdeflate/blob/master/libdeflate.c

That is not something that is supported currently. If it were to be supported, it would need to be tested in CI.

I suppose there wouldn't be too many issues with just concatenating the .c files. However that would not be a real single file amalgamation, since it would still #include all the internal header files. A real single file amalgamation, like the one that SQLite provides, would incorporate all the internal header files into the single .c file too. That would require much more work to support.

What is your use case for a single file amalgamation?

If you are hoping for a performance boost due to function inlining, you'll almost certainly find that there isn't any measurable difference. The different parts of libdeflate are fairly separate from each other, and each function tends to be fairly big.

I do that manual amalgamation so that I could include libdeflate.c (and many other libs the same way) directly to my project. I have like 20 other 3rd party projects that I do this sort of amalgamation. Not perf related. Just don't want to manage separate projects, including src files of them directly works just fine.

I tried to profile my test, libdeflate_deflate_decompress is 45% of the runtime (my test loads thousands of zip files, extracts json file from them, and computes hash of the data). So, that means that actual improvement is much bigger than 25%

Great! Though, I'm surprised that the recent MSVC support improvements would have made libdeflate_deflate_decompress that much faster, as they mainly improved the performance of checksumming. Probably you're actually using libdeflate_zlib_decompress?

I use libdeflate_deflate_decompress directly and I see that in profiler. libdeflate_crc32 takes 2.4% of the cpu in my test (on top of decompress).

I didn't try vs2022, for some reason x64 release build mysteriously fails linking for me.

If it is a problem on libdeflate's side, please report it.

That's not libdeflate related. Some sort of internal but in ms linker.

[Eric Biggers]

I do that manual amalgamation so that I could include libdeflate.c (and many other libs the same way) directly to my project. I have like 20 other 3rd party projects that I do this sort of amalgamation. Not perf related. Just don't want to manage separate projects, including src files of them directly works just fine.

Why not just add the .c files to your project normally?

[Eric Biggers]

Closing this issue. If you'd really like to request support for a single file amalgamation, please create an issue for that.