Giters

Cyan4973 / FiniteStateEntropy

New generation entropy codecs : Finite State Entropy and Huff0

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

huff0 with > 256 symbols

rajputasif opened this issue · comments

commented

I am fascinated by the speed of the huff0 encoder however unfortunately I wish to encode/decode data with 512 bits. I have already tried the fseU16 and it works great, however is there a quick hack/modification I can do to make the huff0 also work with 9-bit data?

Any pointers in this direction are appreciated.
Thanks

UPDATE: My apologies, I didn't realized that HUF_compress2() is doing exactly what I desire. However will have to use it before saying it for sure.

commented

Hi @rajputasif ,

Unfortunately, huff0 wasn't developed for alphabets > 256 symbols,
there is no variant nor hack to trigger it.

Even HUF_compress2() is limited to 256 symbols, it only makes possible to specify a smaller alphabet.

Huffman compressing with an alphabet > 256 symbols is not conceptually impossible,
but will require a substantial rewrite to support this scenario.

commented

Thanks @Cyan4973 for the information and I also realized this after reading the actual source-file.
Actually, I did something similar (i.e. forcing huffman encoding with 32-bit) with the fork of https://github.com/vinx13/huffman in which my hack allowed me to do that.

I agree, the amount of work is definitely substantial (provided that it was not planned beforehead), furthermore since everything is optimized in your code it will be more problematic/buggy for a n00b programmer like me.

Anyways thanks again, I will try to poke around your code in my free time hoping to hit the jackpot.

commented

@Cyan4973 could you please upload the figures from your post(s) http://fastcompression.blogspot.de/2015/07/huffman-revisited-part-1.html ?

I would like to read about the algorithm/implementation in detail before doing anything stupid. If uploading figures is too much hassle, then please send me directly.

Thanks

commented

Do you mean "update the pictures" ?
If you actually mean figures, could you specify which ones need updating ?

commented

I actually meant "upload" as dropbox was saying 404 :)

Ahh they seems to be fixed now (figure for part 3 is still unreachable)... actually I couldn't see them because of some dropbox HREF failed.

Thanks again and my apologies for re-opening cold ones.

commented

Many years ago, Blogspot did not have an included solution to host images.
So I used a dropbox account to host images and created public links to them.

Then, Dropbox abruptly changed its policy for hosted images, resulting in many links broken.
Fortunately, Blogspot now has a proper image hosting service.
So it's mostly a matter of transferring images from one service to another, rebuilding the links, etc.
A tedious process, but it can be done on demand.

commented

Thank you, actually I am not sure if figure (in post3) is important or not as I havent read it yet. I will ask you again in case I couldn't figure out what is written :-)

thanks again and your help is much appreciated.