Improve compression
ALIENQuake opened this issue · comments
Idea description
Please improve the compression. Right now, is worse than warp-packer. Here is some data:
NET Core minimal app folder size: 9 805 053
warp-packer - 5.7 MB
wrappe - 7.7 MB
NET Core Avalonia minimal app folder size: 38 211 348
warp-packer - 16.5 MB
wrappe - 19.1 MB
Additional context
Use-case: distributing three executable (app.exe,app,app.bin) inside one 'package' for Win, Mac and Linux
The file size is the only important factor.
Thanks for the suggestion! Right now this is expected because individual files are compressed separately, and because LZ4 provides a worse compression ratio than DEFLATE. This was chosen so compression and decompression can be as fast as possible and spread over as many cores as available - in fact decompression speed was one of the main motivations behind this project.
It should be easy to add an optional single-threaded compression for the whole data stream. That should help with many small files and still be quite a bit faster than warp. I'll have to compare results to see if that's enough.
Thinking about it some more, compressing the whole stream probably wouldn't affect the results that much and would be inconvenient to implement with the current format.
I've looked into replacing LZ4 with zstd however and the results are promising. I initially chose LZ4 because it provides a much higher theoretical decompression throughput compared to zstd, but in practice the difference seems to be rather slight, at least in this application.
Decompression runtime comparison of a Rust projects target directory (599.88MB, 2602 files), with a LZ4 compression level of 12 (max) and zstd compression level of 12 (of 21), with hyperfine over 20 runs:
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|---|---|---|---|---|
runner-lz4.exe |
2.911 ± 1.856 | 1.602 | 7.709 | 1.00 |
runner-zstd.exe |
2.930 ± 1.338 | 1.760 | 5.630 | 1.01 ± 0.79 |
runner-warp.exe |
15.097 ± 0.248 | 14.710 | 15.565 | 5.19 ± 3.31 |
File sizes of the runners:
runner-lz4.exe 226.03MB
runner-zstd.exe 147.23MB
runner-warp.exe 183.68MB
This is only one datapoint without a fine-tuned compression level, and I'll have to investigate where the huge min/max variation comes from, but it looks like a switch to zstd might be worthwhile.
I'm not quite aware of which 'build' you used for the 'exe-part' which is responsible for running the 'extraction' process and then launching an application but it might be worth mentioning that rust 'releases' (debug/normal) have a significant size difference between them. This is one thing worth checking.
The runners are all built in release mode when wrappe itself is built in release mode:
Lines 86 to 87 in 6fc63e7
Above tests were all done in release mode as well
The runners themselves are very small, so the size difference is almost completely due to the payload compression. Comparison of the runners with an almost empty payload (a 10 byte text file):
runner-lz4-txt.exe 0.37MB
runner-zstd-txt.exe 0.47MB
runner-warp-txt.exe 1.00MB
The zstd version comes with a slightly larger overhead of 100KB compared to the LZ4 version, but the improvement in compression ratio makes that negligible.
The latest snapshot includes the switch to zstd, could you test it out and report back if it brings the expected improvements with your application?
Results:
17.10.2020 18:15 16 650 070 AvApp03-Mini-Win-x32-warp-packer.exe
17.10.2020 18:14 19 192 094 AvApp03-Mini-Win-x32-wrappe-old.exe
17.10.2020 18:14 15 540 294 AvApp03-Mini-Win-x32-wrappe-new.exe
this is fantastic, it's even 1MB smaller than warp-packer. Thank you very much!
Great that it's working well!