Resource framework / PROGMEM compression
yatli opened this issue · comments
@jmamma
https://github.com/siara-cc/Unishox_Arduino_Progmem_lib
We can use this to develop a resource management framework like ".big" files for a game
that's awesome. the decompressor uses up 3000bytes. how much do you think we'll save?
hard to tell, and perhaps it's more effective to concat shorter strings into a big chunk.
I'll give this a try later.
Total compressable object size: 15489
The key property to exploit is that we don't need everything decompressed at all time.
My idea is to allocate a pool to cache these values.
During the setup routine of a page, call the resource manager to pull relevant stuff into the pool.
shox doesn't work for binary data.
https://github.com/pfalcon/uzlib may be a better choice.
cool. we'll find something that works. the principle is sound.
Comes at a good time, I can't enable debug mode anymore.
== Initial comp branch size ==
Sketch uses 228562 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 47314 bytes (83%) of dynamic memory, leaving 9517 bytes for local variables. Maximum is 56831 bytes.
== With uzlib ==
Sketch uses 231026 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 51610 bytes (90%) of dynamic memory, leaving 5221 bytes for local variables. Maximum is 56831 bytes.
PROGMEM +2464
RAM +4296 (4096 buffer)
== Compression run result ==
┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource comp ◇◇ ┆ Yatao
└─╜ ╰▷ .\gen-resource.ps1
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\fonts.cpp
compressed 3621 to 2871 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\icons.cpp
compressed 1986 to 1385 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\machine_names.cpp
compressed 5345 to 2947 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\menu_options.cpp
compressed 738 to 399 raw bytes
Compiling C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource\tuning.cpp
compressed 810 to 700 raw bytes
What do you think?
Note, this framework is general so we can replace gzip with better compressors (as long as a lean decoder exists for arduino)
Brotli destroys gzip:
-a---- 4/11/2021 4:38 AM 2279 fonts.br
-a---- 4/11/2021 4:10 AM 2871 fonts.z
-a---- 4/11/2021 4:38 AM 1085 icons.br
-a---- 4/11/2021 4:10 AM 1385 icons.z
-a---- 4/11/2021 4:38 AM 1521 machine_names.br
-a---- 4/11/2021 4:10 AM 2947 machine_names.z
-a---- 4/11/2021 4:38 AM 314 menu_options.br
-a---- 4/11/2021 4:10 AM 399 menu_options.z
-a---- 4/11/2021 4:38 AM 501 tuning.br
-a---- 4/11/2021 4:10 AM 700 tuning.z
┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource comp ◇◇ ┆ Yatao
└─╜ ╰▷ gci *.hex | Measure-Object -sum -Property Length
Count : 5
Average :
Sum : 12500
Minimum :
Property : Length
┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource comp ◇◇ ┆ Yatao
└─╜ ╰▷ gci *.z | Measure-Object -sum -Property Length
Count : 5
Average :
Sum : 8302
Maximum :
Minimum :
Property : Length
┌─┐ C:\Users\Yatao\Documents\Arduino\hardware\MIDICtrl20_MegaCommand\resource comp ◇◇ ┆ Yatao
└─╜ ╰▷ gci *.br | Measure-Object -sum -Property Length
Count : 5
Average :
Sum : 5700
Maximum :
Minimum :
Property : Length
😱😱😱
net gain unknown, need to port brotli to avr
Decomposing icons.cpp:
About 'Elektrothic'
Can we trim this one down to digits only.
== With unpack ==
Sketch uses 228944 bytes (90%) of program storage space. Maximum is 253952 bytes.
Global variables use 47314 bytes (83%) of dynamic memory, leaving 9517 bytes for local variables. Maximum is 56831 bytes.
Low memory available, stability problems may occur.
PROGMEM +382
RAM +0 (hey)
Awfully low overhead.
About 'Elektrothic'
Can we trim this one down to digits only.
Yes. But we need to identify which bytes in the bitmap structure correspond to the numerals.
comp branch