worst case compression wrongly evaluated
Cyan4973 opened this issue · comments
From Artem Drach :
https://groups.google.com/forum/#!topic/lz4c/SnFzpmyzXKU
Does this bug mean that FSE is not ready for use in a production environment? @Cyan4973 Do you think that 512 bytes should always be enough for the encoded bit stream, and if not, what it should be?
It's a bit early to call FSE "production ready", although that's the plan, and we are slowly getting there. My intention is to move away from "beta status" by the end of the year.
As a sidenote, FSE is getting some good testing through its usage within Zhuff, with several million files correctly compressed/decompressed so far.
(see : https://groups.google.com/forum/#!topic/lz4c/OmA_uSnaO3Q)
Do you think that 512 bytes should always be enough for the encoded bit stream, and if not, what it should be?
If you mean "enough for the header", yes that's what I initially thought.
Unfortunately, Artem tests prove that there is some kind of expansion I've not correctly estimated.
That's something I intent to sort out and correct within the month.
As a sidenote, the issue is only visible when compressing "very" large blocks (such as quite a few megabytes), which is not recommended : in most circumstances, it's much better to compress smaller blocks (such as, typically, 64KB), to ensure proper entropy local adaptation, hence better overall compression ratio.
Thank you, Yann, for your prompt response! This sounds great. Depending on how my compression experiments go this week, we may end up using FSE in our compressed mesh file format. Our largest blocks are 256 KiB. I'll let you know if we encounter any problems!
Latest beta solves this issue.
16fc2e3
Max inflation has been evaluated at 0.45%, plus header.
FSE_compressBound() is a bit of an over-evaluation currently, but it needs to be simple and fast, and it is safer for this function to over-evaluate a bit, rather than under-evaluate as it did up to now; resulting in memory errors.
Thank you!
Committed to master