klauspost / connect-compress

connect-go improved compression

connect-compress

This package provides improved compression schemes for Connect.

Compression is provided from the github.com/klauspost/compress package.

Usage

Import github.com/klauspost/connect-compress/v2.

The compress.WithAll function will return an option that allows both client and servers to compress and decompress all formats.

    // Get the client and server option for all compressors...
    opts := compress.WithAll(compress.LevelBalanced)

    // enable on server
    _, h := pingv1connect.NewPingServiceHandler(&pingServer{}, opts)

    // enable on client
    client := pingv1connect.NewPingServiceClient(http.DefaultClient, url, opts)

By default, the order of preference by the clients is S2, Snappy, Zstandard, Gzip.

To enable client compression and force a specific method use connect.WithSendCompression(...) with one of the 4 provided compression options.

• S2 can be selected for transparent compression, since its performance impact is small.
• Snappy can be used as a more platform independent alternative, with overall less compression.
• Zstandard can be used for efficient compression at good speeds.
• Gzip is the safe fallback.

For more details and options see the documentation.

Note than when OptSmallWindow is used, it must be used on both the client and server.

Supported Formats

For deeper information about the specific implementations see github.com/klauspost/compress project.

All implementations in this package provides very fast handling of incompressible data. That means that it will impose only a minor penalty when sending pre-compressed data.

Compression levels

We expose 3 predefined compression levels.

• LevelFastest will use the fastest available compression level.
• LevelBalanced will use a balanced compression level. Typically half as fast as LevelFastest.
• LevelSmallest will use the strongest and most resource intensive compression method. This is generally not recommended.

General selection criteria:

• Select balanced as your default choice.
• Select the fastest when you expect to transfer large amounts of data locally.
• Select the smallest when transferring via WAN and you have limited bandwidth.

S2

S2 provides better compression than Snappy at similar or better speeds.

Expected performance is ~750MB/s on JSON streams. Size ~2% bigger than gzip on JSON stream.

Approximate speeds on different data types. Compression only, single thread:

	JSON	Binary	Objects	Incompressible
Fastest, MB/s	1145.57	902.15	1054.89	5520.22
Fastest, Reduction	83.40%	66.51%	81.81%	0.00%
Balanced, MB/s	773.94	509.71	721.64	4413.79
Balanced, Reduction	84.79%	69.74%	83.85%	0.00%
Smallest, MB/s	59.66	29.87	58.59	630.93
Smallest, Reduction	86.75%	70.24%	86.30%	0.00%

With OptAllowMultithreadedCompression all cores can used for a roughly linear speed improvement.

Snappy

Snappy uses Google snappy format.

Expected performance is ~600MB/s on JSON streams. Size ~50% bigger than gzip on JSON stream.

Approximate speeds on different data types. Compression only, single thread:

	JSON	Binary	Objects	Incompressible
Fastest, MB/s	1004.25	959.14	1041.04	4511.01
Fastest, Reduction	76.47%	65.79%	75.77%	-0.01%
Balanced, MB/s	599.05	536.50	595.51	2298.54
Balanced, Reduction	77.68%	68.81%	77.46%	-0.01%
Smallest, MB/s	68.50	57.75	70.55	216.17
Smallest, Reduction	78.93%	69.09%	78.58%	-0.01%

With OptAllowMultithreadedCompression all cores can used for a roughly linear speed improvement.

Decompression will usually be limited at around 1000MB/s.

OptSmallWindow has no effect on Snappy, since it is limited to 64KB window already.

Zstandard

Zstandard uses Zstandard compression, but with limited window sizes. Generally Zstandard compresses better and is faster than gzip.

Expected performance is ~300MB/s on JSON streams. Size ~35% smaller than gzip on JSON stream.

Approximate speeds on different data types. Compression only, single thread:

	JSON	Binary	Objects	Incompressible
Fastest, MB/s	611.23	395.09	564.35	2836.57
Fastest, Reduction	88.88%	75.62%	87.76%	0.00%
Balanced, MB/s	322.50	155.23	364.79	2066.60
Balanced, Reduction	90.26%	77.56%	89.33%	0.00%
Smallest, MB/s	36.18	14.42	38.33	172.43
Smallest, Reduction	92.59%	79.40%	91.51%	0.00%

With OptAllowMultithreadedCompression typically 2 goroutines will be used.

Generally decompression should be able to keep up with compression speed.

Gzip

Gzip provides faster compression and decompression methods than the standard library built-in to go-connect.

Expected performance is ~200MB/s on JSON streams. Size reduction is ~85% on JSON stream.

Approximate speeds on different data types. Compression only, single thread:

	JSON	Binary	Objects	Incompressible
Fastest, MB/s	338.17	263.55	373.15	6460.57
Fastest, Reduction	82.14%	75.40%	81.21%	-0.01%
Balanced, MB/s	206.04	148.05	215.19	5535.14
Balanced, Reduction	84.97%	76.64%	83.96%	-0.01%
Smallest, MB/s	59.09	18.21	46.44	119.81
Smallest, Reduction	85.70%	76.60%	85.47%	-0.02%

Note that Gzip decompression speed can often be below 200MB/s, so this will impose the practical limit.

Gzip, stdlib as available in go-connect for reference:

	JSON	Binary	Objects	Incompressible
MB/s	96.05	45.61	89.93	62.74
Reduction	85.61%	76.62%	85.01%	-0.03%

OptSmallWindow has no effect on gzip, since it is limited to a 32KB window already.

License

This package is made available with the Apache License Version 2.0

See LICENSE for more information.