Crusher

High performance, tagged binary data specification like JSON.

Maps directly to and from JSON
Schema less, fully described, like JSON (can be used to store documents)
Little endian for maximum performance on modern CPUs
Efficiently packed, stores bit packed tags and values efficiently
Future proof, supports large numerical types (such as 128 bit integers)

Crusher is much like CBOR or BSON, but is more space efficient in some cases, and is much faster on modern hardware.

Why Tagged Messages?

Flexibility and efficiency

JSON is ubiquitous because it is tagged (has keys), and therefore messages can be sent in part. A fixed binary message without tags means that the entire message must always be sent. Furthermore, extending specifications and adding more fields if far easier with tagged messages and unordered mapping.

Endianness

The endianness must be little endian.

File Extension

The standard extension for crusher files should be .crush

Implementations

C++

Glaze

Right Most Bit Ordering

The right most bit is denoted as the first bit, or bit of index 0.

Compressed Unsigned Integer

A compressed unsigned integer uses the first two bits to denote the number of bytes used to express an integer. The rest of the bits indicate the integer value.

Wherever all caps SIZE is used in the specification, it refers to a size indicator that uses a compressed unsigned integer.

`config` #	Number of Bytes
0	1
1	2
2	4
3	8

Integer Value (N)	Number of Bytes
N < 64 `[2^6]`	1
N < 16384 `[2^14]`	2
N < 1073741824 `[2^30]`	4
N < 4611686018427387904 `[2^62]`	8

Byte Count Indicator

Wherever all caps BYTE COUNT is used, it describes this count indicator.

Mathematically, this is log2(x)

`config` #	Number of bytes
0	1
1	2
2	4
3	8
4	16
5	32
6	64
7	128

Type Header

The type of every value is defined within a single byte. Any unspecified bits must be set to zero.

Wherever all caps HEADER is used, it describes this single byte type header.

The first three bits describe the type via the following numerical values:

0 -> null
1 -> boolean
2 -> number
3 -> string
4 -> object
5 -> typed array
6 -> untyped array
7 -> type tag

Booleans

The next bit of the HEADER indicates true or false.

Numbers

The next two bits of the HEADER indicates whether the number is floating point, signed integer, or unsigned integer.

0 -> floating point
1 -> signed integer
2 -> unsigned integer

The next three bits of the HEADER are used as the BYTE COUNT.

Types conforming to std::is_arithmetic are stored in the same manner as a std::memcpy call on the value.

See Fixed width integer types for integer specification.

int32_t x{};
std::memcpy(destination, &x, sizeof(int32_t));

Strings

The next two bits indicate the number of bytes used for each character (BYTE COUNT).

The transform must be std::memcpy compliant.

Layout: HEADER | SIZE | data_bytes

Objects

The next two bits of the HEADER indicates the type of key.

0 -> string
1 -> signed integer
2 -> unsigned integer

The next three bits of the HEADER indicate the number of bytes used for the integer type (integer keys) or the character type (string keys) (BYTE COUNT).

Typed Arrays

The next two bits indicate the type stored in the array:

0 -> floating point
1 -> signed integer
2 -> unsigned integer
3 -> boolean or string

For integral and floating point types, the next three bits of the type header are the BYTE COUNT.

0 -> boolean
1 -> string

For boolean or string types the next bit indicates whether the type is a boolean or a string. The last two bits are used as the BYTE COUNT for the character size of the string. Boolean arrays are stored using single bits for booleans and packed to the nearest byte.

Layout: HEADER | SIZE | data_bytes

Untyped Arrays

The next three bits of the HEADER are the BYTE COUNT.

Untyped arrays expect elements to have type information.

Type Tag

Expects a subsequent compressed unsigned integer to denote a type tag.

Layout : HEADER | SIZE (i.e. tag) | value

Enums

Enumerations are passed in their integer form.

stephenberry / crusher