Create exact header size instead of defaulting to 127 bytes
simerplaha opened this issue · comments
Simer commented
Overview
A Block
s header size cannot be larger than 127 bytes so for each Block
a header (Array
) get initialised of size 127.
val header = Slice.of[Byte](Byte.MaxValue)
Issue
Even though this byte size (127 bytes) in terms of max file-size (2.GB) is small the in-memory cost of creating Array[Byte]
of size 127 is still expensive.
Solutions
- Find the actual maximum header size required by randomly running tests with various configurations. My guess is that maximum wouldn't be any larger than 60 bytes with minimum being 2 bytes.
- Or Each data-block should provide it's expected header size so header array sizes are exact.