concert / hs-varword

Haskell libraries for variable-length integer encoding

Variable length integer encoding

• attoparsec
• bytestring-builder

This repository consists of two Haskell libraries:

• attoparsec-varword for decoding integers within the attoparsec framework [hackage]
• bytestring-builder-varword for encoding integers via Builders compatible with blaze-builder [hackage]

Motivation

Encoding values of multi-byte types (such as Word32) can be quite wasteful. Often there are many leading zeros which inflate the size of the encoded data. The encodings in this library drop leading empty bytes from the encoding, which produces more compact data.

Broadly speaking, the value to be encoding is divided into 7-bit chunks. The most significant bit of each byte is then used when decoding as a flag to signal whether more data is still to be read for the value being parsed.

This library presents big-endian (most significant byte first) and little-endian (least significant bytes first) variants of the encoding scheme. Further, we include a dense version, used in parts of git, that packs the encoded data a little tighter still.