igxactly / crabz

Like pigz, but rust

🦀 crabz

Like pigz, but rust.

A cross platform, fast, compression and decompression tool.

Synopsis

This is currently a proof of concept CLI tool using the gzp crate.

Supported formats:

• Gzip
• Zlib
• Mgzip
• BGZF
• Raw Deflate
• Snap

Install

• Homebrew / Linuxbrew

brew tap sstadick/crabz
brew install crabz

• Debian (Ubuntu)

curl -LO https://github.com/sstadick/crabz/releases/download/<latest>/crabz-linux-amd64.deb
sudo dpkg -i crabz-linux-amd64.deb

• Cargo

cargo install crabz

• Conda

conda install -c conda-forge crabz

Usage

❯ crabz -h              
Compress and decompress files

USAGE:
    crabz [FLAGS] [OPTIONS] [FILE]

FLAGS:
    -d, --decompress    
            Flag to switch to decompressing inputs. Note: this flag may change in future releases

    -h, --help          
            Prints help information

    -I, --in-place      
            Perform the compression / decompression in place.
            
            **NOTE** this will remove the input file at completion.
    -V, --version       
            Prints version information


OPTIONS:
    -l, --compression-level <compression-level>        
            Compression level [default: 6]

    -p, --compression-threads <compression-threads>
            Number of compression threads to use, or if decompressing a format that allow for multi-threaded
            decompression, the number to use. Note that > 4 threads for decompression doesn't seem to help [default:
            32]
    -f, --format <format>
            The format to use [default: gzip]  [possible values: gzip, bgzf, mgzip,
            zlib, deflate, snap]
    -o, --output <output>                              
            Output path to write to, empty or "-" to write to stdout

    -P, --pin-at <pin-at>                              
            Specify the physical core to pin threads at.
            
            This can provide a significant performance improvement, but has the downside of possibly conflicting with
            other pinned cores. If you are running multiple instances of `crabz` at once you can manually space out the
            pinned cores.
            
            # Example
            - Instance 1 has `-p 4 -P 0` set indicating that it will use 4 cores pinned at 0, 1, 2, 3
            - Instance 2 has `-p 4 -P 4` set indicating that it will use 4 cores pinned at 4, 5, 6, 7

ARGS:
    <FILE>    
            Input file to read from, empty or "-" to read from stdin

Benchmarks

These benchmarks use the data in bench-data catted together 100 times. Run with bash ./benchmark.sh data.txt.

Benchmark system specs: Ubuntu 20 AMD Ryzen 9 3950X 16-Core Processor w/ 64 GB DDR4 memory and 1TB NVMe Drive

pigz v2.4 installed via apt on Ubuntu

Takeaways:

• crabz with zlib backend is pretty much identical to pigz
• crabz with zlib-ng backend is roughly 30-50% faster than pigz
• crabz with rust backend is roughly 5-10% faster than pigz

It is already known that zlib-ng is faster than zlib, so none of this is groundbreaking. However, I think crabz gets an an edge due to the following:

• crabz with deflate_rust backend is using all Rust only code, which is in theory more secure / safe.
• crabz with zlib-ng is easier to install than pigz with a zlib-ng backend
• crabz supports more formats than pigz
• crabz is cross platform and can run on windows

With regards to block formats like Mgzip and BGZF, crabz is using libdeflater by default which excels at compressing and decompression known-sized blocks. This makes block compression formats very fast at a small loss to the compression ratio.

See end of benchmarks section for comparison against bgzip.

As crabz is just a wrapper for the gzp library, the most exciting thing about these benchmarks is that gzp is on par with best in class CLI tools for multi-threaded compression and decompression as a library.

Flate2 zlib-ng backend

Compression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -p 1 -c 3 < ./data.txt	6.450 ± 0.069	6.328	6.540	16.86 ± 0.24
pigz -p 1 -3 < ./data.txt	11.404 ± 0.152	11.186	11.717	29.81 ± 0.49
crabz -p 2 -c 3 < ./data.txt	3.437 ± 0.017	3.418	3.461	8.98 ± 0.10
pigz -p 2 -3 < ./data.txt	5.868 ± 0.031	5.826	5.927	15.34 ± 0.17
crabz -p 4 -c 3 < ./data.txt	1.741 ± 0.008	1.729	1.752	4.55 ± 0.05
pigz -p 4 -3 < ./data.txt	2.952 ± 0.008	2.939	2.960	7.72 ± 0.08
crabz -p 8 -c 3 < ./data.txt	0.889 ± 0.004	0.882	0.895	2.32 ± 0.02
pigz -p 8 -3 < ./data.txt	1.505 ± 0.008	1.493	1.520	3.93 ± 0.04
crabz -p 16 -c 3 < ./data.txt	0.485 ± 0.014	0.457	0.502	1.27 ± 0.04
pigz -p 16 -3 < ./data.txt	0.775 ± 0.011	0.764	0.797	2.02 ± 0.04
crabz -p 32 -c 3 < ./data.txt	0.383 ± 0.004	0.375	0.388	1.00
pigz -p 32 -3 < ./data.txt	0.699 ± 0.029	0.668	0.770	1.83 ± 0.08
crabz -p 1 -c 6 < ./data.txt	10.367 ± 0.211	10.106	10.642	27.10 ± 0.61
pigz -p 1 -6 < ./data.txt	26.734 ± 0.345	26.234	27.135	69.89 ± 1.12
crabz -p 2 -c 6 < ./data.txt	5.366 ± 0.036	5.299	5.429	14.03 ± 0.16
pigz -p 2 -6 < ./data.txt	13.589 ± 0.083	13.428	13.679	35.52 ± 0.40
crabz -p 4 -c 6 < ./data.txt	2.719 ± 0.021	2.694	2.757	7.11 ± 0.09
pigz -p 4 -6 < ./data.txt	6.887 ± 0.013	6.871	6.916	18.00 ± 0.17
crabz -p 8 -c 6 < ./data.txt	1.381 ± 0.007	1.372	1.397	3.61 ± 0.04
pigz -p 8 -6 < ./data.txt	3.479 ± 0.008	3.463	3.488	9.09 ± 0.09
crabz -p 16 -c 6 < ./data.txt	0.745 ± 0.022	0.727	0.804	1.95 ± 0.06
pigz -p 16 -6 < ./data.txt	1.818 ± 0.036	1.765	1.874	4.75 ± 0.10
crabz -p 32 -c 6 < ./data.txt	0.549 ± 0.006	0.538	0.557	1.44 ± 0.02
pigz -p 32 -6 < ./data.txt	1.187 ± 0.011	1.172	1.210	3.10 ± 0.04
crabz -p 1 -c 9 < ./data.txt	30.114 ± 0.196	29.842	30.420	78.72 ± 0.90
pigz -p 1 -9 < ./data.txt	51.369 ± 0.164	51.246	51.698	134.29 ± 1.33
crabz -p 2 -c 9 < ./data.txt	15.371 ± 0.070	15.202	15.443	40.18 ± 0.42
pigz -p 2 -9 < ./data.txt	26.452 ± 0.085	26.253	26.576	69.15 ± 0.69
crabz -p 4 -c 9 < ./data.txt	7.729 ± 0.022	7.699	7.768	20.20 ± 0.20
pigz -p 4 -9 < ./data.txt	13.365 ± 0.047	13.271	13.449	34.94 ± 0.35
crabz -p 8 -c 9 < ./data.txt	3.901 ± 0.006	3.889	3.910	10.20 ± 0.10
pigz -p 8 -9 < ./data.txt	6.749 ± 0.014	6.737	6.781	17.64 ± 0.17
crabz -p 16 -c 9 < ./data.txt	2.039 ± 0.024	1.997	2.071	5.33 ± 0.08
pigz -p 16 -9 < ./data.txt	3.486 ± 0.054	3.426	3.574	9.11 ± 0.17
crabz -p 32 -c 9 < ./data.txt	1.337 ± 0.072	1.220	1.411	3.49 ± 0.19
pigz -p 32 -9 < ./data.txt	2.203 ± 0.114	2.082	2.378	5.76 ± 0.30

Decompression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -d < ./data.3.txt.gz	1.422 ± 0.010	1.411	1.437	1.03 ± 0.02
pigz -d < ./data.3.txt.gz	1.674 ± 0.031	1.621	1.705	1.21 ± 0.03
crabz -d < ./data.6.txt.gz	1.403 ± 0.016	1.389	1.427	1.01 ± 0.02
pigz -d < ./data.6.txt.gz	1.724 ± 0.026	1.697	1.766	1.24 ± 0.02
crabz -d < ./data.9.txt.gz	1.385 ± 0.018	1.359	1.416	1.00
pigz -d < ./data.9.txt.gz	1.745 ± 0.044	1.684	1.797	1.26 ± 0.04

Flate2 zlib backend

Compression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -p 1 -c 3 < ./data.txt	11.248 ± 0.247	11.085	11.532	20.23 ± 0.45
pigz -p 1 -3 < ./data.txt	11.296 ± 0.171	11.104	11.434	20.32 ± 0.31
crabz -p 2 -c 3 < ./data.txt	5.681 ± 0.040	5.645	5.725	10.22 ± 0.08
pigz -p 2 -3 < ./data.txt	5.926 ± 0.015	5.916	5.944	10.66 ± 0.04
crabz -p 4 -c 3 < ./data.txt	2.891 ± 0.007	2.883	2.895	5.20 ± 0.02
pigz -p 4 -3 < ./data.txt	2.966 ± 0.013	2.955	2.980	5.34 ± 0.03
crabz -p 8 -c 3 < ./data.txt	1.461 ± 0.003	1.459	1.465	2.63 ± 0.01
pigz -p 8 -3 < ./data.txt	1.509 ± 0.004	1.505	1.512	2.71 ± 0.01
crabz -p 16 -c 3 < ./data.txt	0.784 ± 0.010	0.775	0.795	1.41 ± 0.02
pigz -p 16 -3 < ./data.txt	0.772 ± 0.010	0.765	0.784	1.39 ± 0.02
crabz -p 32 -c 3 < ./data.txt	0.556 ± 0.002	0.554	0.557	1.00
pigz -p 32 -3 < ./data.txt	0.743 ± 0.047	0.694	0.786	1.34 ± 0.08
crabz -p 1 -c 6 < ./data.txt	26.366 ± 0.154	26.189	26.469	47.42 ± 0.31
pigz -p 1 -6 < ./data.txt	26.688 ± 0.103	26.579	26.783	48.00 ± 0.23
crabz -p 2 -c 6 < ./data.txt	13.443 ± 0.069	13.400	13.523	24.18 ± 0.14
pigz -p 2 -6 < ./data.txt	13.605 ± 0.059	13.567	13.673	24.47 ± 0.13
crabz -p 4 -c 6 < ./data.txt	6.833 ± 0.005	6.828	6.837	12.29 ± 0.03
pigz -p 4 -6 < ./data.txt	6.866 ± 0.028	6.834	6.884	12.35 ± 0.06
crabz -p 8 -c 6 < ./data.txt	3.446 ± 0.000	3.445	3.446	6.20 ± 0.02
pigz -p 8 -6 < ./data.txt	3.482 ± 0.002	3.480	3.483	6.26 ± 0.02
crabz -p 16 -c 6 < ./data.txt	1.822 ± 0.012	1.813	1.835	3.28 ± 0.02
pigz -p 16 -6 < ./data.txt	1.771 ± 0.004	1.767	1.776	3.19 ± 0.01
crabz -p 32 -c 6 < ./data.txt	1.178 ± 0.008	1.171	1.187	2.12 ± 0.02
pigz -p 32 -6 < ./data.txt	1.184 ± 0.001	1.184	1.185	2.13 ± 0.01
crabz -p 1 -c 9 < ./data.txt	52.122 ± 0.288	51.790	52.293	93.75 ± 0.58
pigz -p 1 -9 < ./data.txt	53.031 ± 0.071	52.951	53.085	95.39 ± 0.29
crabz -p 2 -c 9 < ./data.txt	26.287 ± 0.047	26.249	26.339	47.28 ± 0.15
pigz -p 2 -9 < ./data.txt	26.409 ± 0.238	26.190	26.662	47.50 ± 0.45
crabz -p 4 -c 9 < ./data.txt	13.373 ± 0.051	13.317	13.419	24.05 ± 0.11
pigz -p 4 -9 < ./data.txt	13.414 ± 0.035	13.383	13.451	24.13 ± 0.09
crabz -p 8 -c 9 < ./data.txt	6.733 ± 0.003	6.731	6.736	12.11 ± 0.03
pigz -p 8 -9 < ./data.txt	6.763 ± 0.004	6.761	6.767	12.16 ± 0.03
crabz -p 16 -c 9 < ./data.txt	3.487 ± 0.034	3.450	3.517	6.27 ± 0.06
pigz -p 16 -9 < ./data.txt	3.459 ± 0.021	3.434	3.473	6.22 ± 0.04
crabz -p 32 -c 9 < ./data.txt	2.088 ± 0.008	2.081	2.097	3.76 ± 0.02
pigz -p 32 -9 < ./data.txt	2.107 ± 0.023	2.090	2.133	3.79 ± 0.04

Decompression

Flate2 rust backend

Compression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -p 1 -c 3 < ./data.txt	10.167 ± 0.164	10.050	10.355	18.57 ± 0.33
pigz -p 1 -3 < ./data.txt	11.338 ± 0.071	11.292	11.420	20.71 ± 0.21
crabz -p 2 -c 3 < ./data.txt	4.912 ± 0.013	4.898	4.920	8.97 ± 0.08
pigz -p 2 -3 < ./data.txt	5.876 ± 0.047	5.826	5.919	10.73 ± 0.12
crabz -p 4 -c 3 < ./data.txt	2.463 ± 0.018	2.447	2.482	4.50 ± 0.05
pigz -p 4 -3 < ./data.txt	2.967 ± 0.008	2.958	2.972	5.42 ± 0.05
crabz -p 8 -c 3 < ./data.txt	1.255 ± 0.005	1.250	1.261	2.29 ± 0.02
pigz -p 8 -3 < ./data.txt	1.509 ± 0.002	1.507	1.511	2.76 ± 0.02
crabz -p 16 -c 3 < ./data.txt	0.705 ± 0.030	0.673	0.731	1.29 ± 0.05
pigz -p 16 -3 < ./data.txt	0.780 ± 0.015	0.768	0.797	1.42 ± 0.03
crabz -p 32 -c 3 < ./data.txt	0.547 ± 0.004	0.544	0.552	1.00
pigz -p 32 -3 < ./data.txt	0.755 ± 0.025	0.726	0.771	1.38 ± 0.05
crabz -p 1 -c 6 < ./data.txt	27.064 ± 0.288	26.863	27.394	49.44 ± 0.66
pigz -p 1 -6 < ./data.txt	27.034 ± 0.090	26.938	27.117	49.38 ± 0.43
crabz -p 2 -c 6 < ./data.txt	12.400 ± 0.083	12.321	12.487	22.65 ± 0.24
pigz -p 2 -6 < ./data.txt	13.619 ± 0.074	13.558	13.702	24.88 ± 0.24
crabz -p 4 -c 6 < ./data.txt	6.279 ± 0.023	6.263	6.305	11.47 ± 0.10
pigz -p 4 -6 < ./data.txt	6.879 ± 0.020	6.867	6.901	12.57 ± 0.11
crabz -p 8 -c 6 < ./data.txt	3.189 ± 0.010	3.178	3.198	5.83 ± 0.05
pigz -p 8 -6 < ./data.txt	3.477 ± 0.007	3.470	3.483	6.35 ± 0.05
crabz -p 16 -c 6 < ./data.txt	1.756 ± 0.015	1.740	1.771	3.21 ± 0.04
pigz -p 16 -6 < ./data.txt	1.799 ± 0.024	1.779	1.827	3.29 ± 0.05
crabz -p 32 -c 6 < ./data.txt	1.192 ± 0.011	1.183	1.205	2.18 ± 0.03
pigz -p 32 -6 < ./data.txt	1.196 ± 0.016	1.183	1.214	2.19 ± 0.03
crabz -p 1 -c 9 < ./data.txt	44.907 ± 0.283	44.585	45.116	82.03 ± 0.84
pigz -p 1 -9 < ./data.txt	53.109 ± 1.049	52.373	54.311	97.02 ± 2.07
crabz -p 2 -c 9 < ./data.txt	19.977 ± 0.159	19.819	20.136	36.49 ± 0.41
pigz -p 2 -9 < ./data.txt	26.562 ± 0.134	26.407	26.643	48.52 ± 0.46
crabz -p 4 -c 9 < ./data.txt	10.397 ± 0.484	10.070	10.953	18.99 ± 0.90
pigz -p 4 -9 < ./data.txt	13.346 ± 0.040	13.300	13.372	24.38 ± 0.21
crabz -p 8 -c 9 < ./data.txt	5.100 ± 0.021	5.076	5.114	9.32 ± 0.08
pigz -p 8 -9 < ./data.txt	6.754 ± 0.016	6.736	6.767	12.34 ± 0.10
crabz -p 16 -c 9 < ./data.txt	2.716 ± 0.014	2.708	2.732	4.96 ± 0.05
pigz -p 16 -9 < ./data.txt	3.444 ± 0.038	3.420	3.487	6.29 ± 0.09
crabz -p 32 -c 9 < ./data.txt	1.747 ± 0.009	1.740	1.758	3.19 ± 0.03
pigz -p 32 -9 < ./data.txt	2.086 ± 0.008	2.077	2.093	3.81 ± 0.03

Decompression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -d < ./data.3.txt.gz	1.599 ± 0.014	1.573	1.615	1.00
pigz -d < ./data.3.txt.gz	1.696 ± 0.020	1.654	1.725	1.06 ± 0.02
crabz -d < ./data.6.txt.gz	1.615 ± 0.012	1.586	1.626	1.01 ± 0.01
pigz -d < ./data.6.txt.gz	1.760 ± 0.030	1.687	1.797	1.10 ± 0.02
crabz -d < ./data.9.txt.gz	1.613 ± 0.014	1.596	1.641	1.01 ± 0.01
pigz -d < ./data.9.txt.gz	1.767 ± 0.012	1.748	1.787	1.11 ± 0.01

Block Formats with libdeflater

Decompression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -p 1 -d -f mgzip ./bdata.3.txt.gz > data.txt	1.221 ± 0.164	1.073	1.397	2.32 ± 0.31
pigz -d -c ./bdata.3.txt.gz > data.txt	2.415 ± 0.063	2.347	2.472	4.58 ± 0.14
crabz -p 1 -d -f mgzip ./bdata.6.txt.gz > data.txt	1.256 ± 0.063	1.200	1.325	2.38 ± 0.13
pigz -d -c ./bdata.6.txt.gz > data.txt	2.513 ± 0.052	2.467	2.569	4.77 ± 0.13
crabz -p 1 -d -f mgzip ./bdata.9.txt.gz > data.txt	1.147 ± 0.065	1.094	1.219	2.18 ± 0.13
pigz -d -c ./bdata.9.txt.gz > data.txt	2.394 ± 0.118	2.262	2.488	4.54 ± 0.24
crabz -p 1 -d -f mgzip ./bdata.12.txt.gz > data.txt	1.165 ± 0.074	1.106	1.248	2.21 ± 0.15
pigz -d -c ./bdata.12.txt.gz > data.txt	2.457 ± 0.067	2.408	2.534	4.66 ± 0.15
crabz -p 2 -d -f mgzip ./bdata.3.txt.gz > data.txt	0.634 ± 0.008	0.628	0.642	1.20 ± 0.03
pigz -d -c ./bdata.3.txt.gz > data.txt	2.379 ± 0.012	2.368	2.391	4.51 ± 0.08
crabz -p 2 -d -f mgzip ./bdata.6.txt.gz > data.txt	0.645 ± 0.015	0.629	0.658	1.22 ± 0.03
pigz -d -c ./bdata.6.txt.gz > data.txt	2.438 ± 0.073	2.356	2.497	4.62 ± 0.16
crabz -p 2 -d -f mgzip ./bdata.9.txt.gz > data.txt	0.659 ± 0.015	0.644	0.674	1.25 ± 0.04
pigz -d -c ./bdata.9.txt.gz > data.txt	2.451 ± 0.075	2.400	2.538	4.65 ± 0.16
crabz -p 2 -d -f mgzip ./bdata.12.txt.gz > data.txt	0.656 ± 0.015	0.647	0.673	1.24 ± 0.04
pigz -d -c ./bdata.12.txt.gz > data.txt	2.450 ± 0.045	2.412	2.500	4.65 ± 0.12
crabz -p 4 -d -f mgzip ./bdata.3.txt.gz > data.txt	0.577 ± 0.024	0.554	0.603	1.10 ± 0.05
pigz -d -c ./bdata.3.txt.gz > data.txt	2.459 ± 0.052	2.420	2.518	4.66 ± 0.13
crabz -p 4 -d -f mgzip ./bdata.6.txt.gz > data.txt	0.559 ± 0.024	0.531	0.576	1.06 ± 0.05
pigz -d -c ./bdata.6.txt.gz > data.txt	2.538 ± 0.044	2.502	2.587	4.81 ± 0.12
crabz -p 4 -d -f mgzip ./bdata.9.txt.gz > data.txt	0.552 ± 0.011	0.539	0.560	1.05 ± 0.03
pigz -d -c ./bdata.9.txt.gz > data.txt	2.402 ± 0.018	2.385	2.420	4.56 ± 0.08
crabz -p 4 -d -f mgzip ./bdata.12.txt.gz > data.txt	0.592 ± 0.040	0.546	0.616	1.12 ± 0.08
pigz -d -c ./bdata.12.txt.gz > data.txt	2.525 ± 0.038	2.484	2.558	4.79 ± 0.11
crabz -p 8 -d -f mgzip ./bdata.3.txt.gz > data.txt	0.563 ± 0.013	0.548	0.571	1.07 ± 0.03
pigz -d -c ./bdata.3.txt.gz > data.txt	2.490 ± 0.126	2.369	2.621	4.72 ± 0.25
crabz -p 8 -d -f mgzip ./bdata.6.txt.gz > data.txt	0.552 ± 0.018	0.533	0.569	1.05 ± 0.04
pigz -d -c ./bdata.6.txt.gz > data.txt	2.531 ± 0.115	2.417	2.647	4.80 ± 0.23
crabz -p 8 -d -f mgzip ./bdata.9.txt.gz > data.txt	0.603 ± 0.029	0.583	0.636	1.14 ± 0.06
pigz -d -c ./bdata.9.txt.gz > data.txt	2.483 ± 0.042	2.435	2.515	4.71 ± 0.11
crabz -p 8 -d -f mgzip ./bdata.12.txt.gz > data.txt	0.527 ± 0.009	0.519	0.537	1.00
pigz -d -c ./bdata.12.txt.gz > data.txt	2.524 ± 0.093	2.417	2.583	4.79 ± 0.19
crabz -p 16 -d -f mgzip ./bdata.3.txt.gz > data.txt	0.603 ± 0.058	0.551	0.665	1.14 ± 0.11
pigz -d -c ./bdata.3.txt.gz > data.txt	2.392 ± 0.007	2.384	2.397	4.54 ± 0.08
crabz -p 16 -d -f mgzip ./bdata.6.txt.gz > data.txt	0.611 ± 0.065	0.565	0.686	1.16 ± 0.13
pigz -d -c ./bdata.6.txt.gz > data.txt	2.593 ± 0.148	2.427	2.712	4.92 ± 0.29
crabz -p 16 -d -f mgzip ./bdata.9.txt.gz > data.txt	0.564 ± 0.027	0.541	0.594	1.07 ± 0.05
pigz -d -c ./bdata.9.txt.gz > data.txt	2.426 ± 0.023	2.404	2.450	4.60 ± 0.09
crabz -p 16 -d -f mgzip ./bdata.12.txt.gz > data.txt	0.601 ± 0.020	0.582	0.623	1.14 ± 0.04
pigz -d -c ./bdata.12.txt.gz > data.txt	2.528 ± 0.022	2.507	2.550	4.80 ± 0.09
crabz -p 32 -d -f mgzip ./bdata.3.txt.gz > data.txt	0.595 ± 0.019	0.577	0.614	1.13 ± 0.04
pigz -d -c ./bdata.3.txt.gz > data.txt	2.544 ± 0.107	2.422	2.621	4.83 ± 0.22
crabz -p 32 -d -f mgzip ./bdata.6.txt.gz > data.txt	0.601 ± 0.021	0.586	0.626	1.14 ± 0.05
pigz -d -c ./bdata.6.txt.gz > data.txt	2.519 ± 0.114	2.435	2.649	4.78 ± 0.23
crabz -p 32 -d -f mgzip ./bdata.9.txt.gz > data.txt	0.565 ± 0.023	0.539	0.579	1.07 ± 0.05
pigz -d -c ./bdata.9.txt.gz > data.txt	2.487 ± 0.064	2.415	2.540	4.72 ± 0.15
crabz -p 32 -d -f mgzip ./bdata.12.txt.gz > data.txt	0.557 ± 0.013	0.548	0.571	1.06 ± 0.03
pigz -d -c ./bdata.12.txt.gz > data.txt	2.505 ± 0.105	2.442	2.626	4.75 ± 0.22

crabz, pigz, and bgzip

These benchmarks were run on the all_train.csv data found here

Compression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -p 2 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz	15.837 ± 0.137	15.688	15.959	5.52 ± 0.13
bgzip -c -@ 2 -l 2 ./data.txt > ./data.out.txt.gz	19.471 ± 0.178	19.268	19.602	6.78 ± 0.16
crabz -p 2 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz	19.723 ± 0.632	19.285	20.448	6.87 ± 0.26
pigz -c -p 2 -2 ./data.txt > ./data.out.txt.gz	32.249 ± 0.024	32.226	32.274	11.24 ± 0.24
crabz -p 4 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz	8.601 ± 0.538	8.040	9.113	3.00 ± 0.20
bgzip -c -@ 4 -l 2 ./data.txt > ./data.out.txt.gz	10.953 ± 0.033	10.929	10.990	3.82 ± 0.08
crabz -p 4 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz	10.887 ± 0.584	10.236	11.364	3.79 ± 0.22
pigz -c -p 4 -2 ./data.txt > ./data.out.txt.gz	16.493 ± 0.323	16.257	16.861	5.75 ± 0.17
crabz -p 8 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz	5.206 ± 0.372	4.780	5.464	1.81 ± 0.14
bgzip -c -@ 8 -l 2 ./data.txt > ./data.out.txt.gz	6.920 ± 0.033	6.893	6.957	2.41 ± 0.05
crabz -p 8 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz	5.893 ± 0.135	5.777	6.041	2.05 ± 0.06
pigz -c -p 8 -2 ./data.txt > ./data.out.txt.gz	8.974 ± 0.467	8.553	9.477	3.13 ± 0.18
crabz -p 16 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz	2.870 ± 0.061	2.816	2.936	1.00
bgzip -c -@ 16 -l 2 ./data.txt > ./data.out.txt.gz	5.124 ± 0.107	5.040	5.244	1.79 ± 0.05
crabz -p 16 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz	4.250 ± 0.323	3.933	4.579	1.48 ± 0.12
pigz -c -p 16 -2 ./data.txt > ./data.out.txt.gz	4.767 ± 0.223	4.513	4.933	1.66 ± 0.09
crabz -p 32 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz	3.669 ± 0.303	3.320	3.865	1.28 ± 0.11
bgzip -c -@ 32 -l 2 ./data.txt > ./data.out.txt.gz	4.676 ± 0.038	4.632	4.701	1.63 ± 0.04
crabz -p 32 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz	4.324 ± 0.246	4.143	4.605	1.51 ± 0.09
pigz -c -p 32 -2 ./data.txt > ./data.out.txt.gz	5.854 ± 0.070	5.795	5.931	2.04 ± 0.05
crabz -p 2 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz	27.696 ± 0.147	27.593	27.864	9.65 ± 0.21
bgzip -c -@ 2 -l 6 ./data.txt > ./data.out.txt.gz	30.961 ± 0.446	30.446	31.231	10.79 ± 0.28
crabz -p 2 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz	36.229 ± 0.175	36.092	36.427	12.62 ± 0.27
pigz -c -p 2 -6 ./data.txt > ./data.out.txt.gz	97.175 ± 0.571	96.743	97.823	33.86 ± 0.74
crabz -p 4 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz	14.802 ± 0.436	14.316	15.159	5.16 ± 0.19
bgzip -c -@ 4 -l 6 ./data.txt > ./data.out.txt.gz	16.927 ± 0.130	16.789	17.048	5.90 ± 0.13
crabz -p 4 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz	19.192 ± 0.675	18.629	19.940	6.69 ± 0.27
pigz -c -p 4 -6 ./data.txt > ./data.out.txt.gz	49.305 ± 0.114	49.203	49.429	17.18 ± 0.37
crabz -p 8 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz	7.833 ± 0.065	7.784	7.907	2.73 ± 0.06
bgzip -c -@ 8 -l 6 ./data.txt > ./data.out.txt.gz	9.858 ± 0.105	9.739	9.939	3.43 ± 0.08
crabz -p 8 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz	10.417 ± 0.979	9.626	11.511	3.63 ± 0.35
pigz -c -p 8 -6 ./data.txt > ./data.out.txt.gz	25.276 ± 0.170	25.083	25.404	8.81 ± 0.20
crabz -p 16 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz	4.704 ± 0.321	4.337	4.937	1.64 ± 0.12
bgzip -c -@ 16 -l 6 ./data.txt > ./data.out.txt.gz	6.565 ± 0.155	6.429	6.734	2.29 ± 0.07
crabz -p 16 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz	5.722 ± 0.320	5.530	6.092	1.99 ± 0.12
pigz -c -p 16 -6 ./data.txt > ./data.out.txt.gz	13.673 ± 0.129	13.525	13.762	4.76 ± 0.11
crabz -p 32 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz	4.202 ± 0.213	3.957	4.328	1.46 ± 0.08
bgzip -c -@ 32 -l 6 ./data.txt > ./data.out.txt.gz	5.538 ± 0.135	5.395	5.663	1.93 ± 0.06
crabz -p 32 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz	5.488 ± 0.064	5.423	5.550	1.91 ± 0.05
pigz -c -p 32 -6 ./data.txt > ./data.out.txt.gz	9.079 ± 0.286	8.808	9.379	3.16 ± 0.12
crabz -p 2 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz	162.875 ± 0.100	162.778	162.977	56.75 ± 1.20
bgzip -c -@ 2 -l 9 ./data.txt > ./data.out.txt.gz	172.428 ± 0.242	172.207	172.687	60.08 ± 1.27
crabz -p 2 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz	139.245 ± 0.270	138.974	139.514	48.52 ± 1.03
pigz -c -p 2 -9 ./data.txt > ./data.out.txt.gz	209.645 ± 0.058	209.580	209.691	73.05 ± 1.55
crabz -p 4 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz	84.624 ± 0.185	84.414	84.762	29.49 ± 0.63
bgzip -c -@ 4 -l 9 ./data.txt > ./data.out.txt.gz	87.228 ± 0.232	87.053	87.492	30.39 ± 0.65
crabz -p 4 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz	72.339 ± 0.166	72.187	72.517	25.21 ± 0.54
pigz -c -p 4 -9 ./data.txt > ./data.out.txt.gz	106.579 ± 0.236	106.307	106.731	37.14 ± 0.79
crabz -p 8 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz	42.988 ± 0.130	42.905	43.138	14.98 ± 0.32
bgzip -c -@ 8 -l 9 ./data.txt > ./data.out.txt.gz	44.550 ± 0.097	44.449	44.642	15.52 ± 0.33
crabz -p 8 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz	36.555 ± 0.030	36.521	36.579	12.74 ± 0.27
pigz -c -p 8 -9 ./data.txt > ./data.out.txt.gz	54.047 ± 0.016	54.030	54.062	18.83 ± 0.40
crabz -p 16 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz	22.391 ± 0.234	22.154	22.623	7.80 ± 0.18
bgzip -c -@ 16 -l 9 ./data.txt > ./data.out.txt.gz	24.041 ± 0.237	23.813	24.286	8.38 ± 0.20
crabz -p 16 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz	19.285 ± 0.125	19.141	19.363	6.72 ± 0.15
pigz -c -p 16 -9 ./data.txt > ./data.out.txt.gz	27.645 ± 0.078	27.579	27.731	9.63 ± 0.21
crabz -p 32 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz	15.148 ± 0.138	14.992	15.252	5.28 ± 0.12
bgzip -c -@ 32 -l 9 ./data.txt > ./data.out.txt.gz	16.091 ± 0.193	15.874	16.243	5.61 ± 0.14
crabz -p 32 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz	11.832 ± 0.168	11.637	11.930	4.12 ± 0.11
pigz -c -p 32 -9 ./data.txt > ./data.out.txt.gz	16.912 ± 0.095	16.804	16.982	5.89 ± 0.13

Decompression

Command	Mean [s]	Min [s]	Max [s]	Relative
crabz -d -p 4 -f bgzf ./data.txt.gz > ./data.out.txt	5.941 ± 0.172	5.745	6.070	1.11 ± 0.09
bgzip -d -c -@ 4 ./data.txt.gz > ./data.out.txt	5.357 ± 0.407	4.925	5.734	1.00
crabz -d -p 8 -f bgzf ./data.txt.gz > ./data.out.txt	5.569 ± 0.496	5.023	5.990	1.04 ± 0.12
bgzip -d -c -@ 8 ./data.txt.gz > ./data.out.txt	5.867 ± 0.252	5.682	6.154	1.10 ± 0.10
crabz -d -p 16 -f bgzf ./data.txt.gz > ./data.out.txt	5.663 ± 0.240	5.506	5.939	1.06 ± 0.09
bgzip -d -c -@ 16 ./data.txt.gz > ./data.out.txt	5.534 ± 0.124	5.416	5.663	1.03 ± 0.08

TODOs

• Add some form of auto format detection, even just by file extension