Question: does Nippy Zstandard support training?
lowecg opened this issue · comments
I see that Zstandard compression support has been added, which is great news.
We use Nippy to generate lots of small documents with similar data, which seems like a good fit for Zstandard training. Is training possible with Nippy?
@lowecg Hi there!
Is training possible with Nippy?
Not currently, at least. Nippy v3.4 uses aircompressor for its compression - and from a quick skim of the docs, I don't believe that it currently offers any form of explicit/opt-in training.
I'm not too familiar with Zstd myself, or its training/dictionary features.
Do you have any idea what happens if a particular dictionary is used during compression, but then unavailable for decompression?
BTW it's worth noting that you can of course pass Nippy's uncompressed output to any compression tool you like. The built in compression schemes are there mostly for convenience.
There's some performance+ratio bench info on the current out-the-box compressors available here in case that's helpful.
Cheers!