S2 compression
franchb opened this issue · comments
According to recent post of @klauspost on Reddit:
S2 is meant to be the choice for absolute speed. While it is only available in Go other languages can simply use Snappy for compression and implementing the decoder changes should be quite trivial, since the changes are very small.
For the rest I would probably go for zstd - especially if I can find the time to implement even stronger compression for the Go version.
S2, Go data compression at GBs per second
Maybe it could be implemented in taskq?
It should be a fine match since it can pretty much always compress at IO speeds.
I don't mind supporting another compression method - it is only a matter of time required to implement it and I don't have much of it...
But I wonder why do you ask - is zstd too slow or is it a cross compilation problem?
In project I work on S2 shows slightly better IO performance, so I already use it as an option (s2, zstd, etc, thanks to the excellent libraries of @klauspost ).
I will try to implement PR if you don't mind.
@franchb s2 is welcome for sure. But please note that I've just pushed v3 (it is working but API not stable). So if you plan to make changes - please make sure you have the latest version.
Thank you! I will start with v3.
This is closed by 2b763bb.
s2 now is the default since it is 2-3x faster especially for small payloads up to 256kb (which is the SQS limit). Comperssion is only 5%-10% worse than zstd.