dechristopher / lod

LOD is an intelligent map tile caching proxy for the edge.

Home Page:https://lod.tile.fund

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[v1.0.0] Build binary data structure for HTTP headers and tile

dechristopher opened this issue · comments

Since we're migrating to bigcache and want to use Redis soon, we'll need to be able to serialize a tile and its saved HTTP headers into a binary data structure. This will allow us to stray from using JSON or other formats that aren't efficient to process. Luckily, all of our tile data is either binary serialized protocol buffers or raw raster image data so we don't need to worry much about processing anything but the few bytes prepended to the tile data for the HTTP headers. The implementation should support header values of arbitrary, yet sane length.

Global counter Num bytes, H1 key H1 key data Num bytes, H1 value H1 value data Tile data ...
1 byte - uint8 2 bytes - uint16 N bytes 2 bytes - uint16 N bytes N bytes
1 6 Cached 4 miss ...

The deserializer will read n headers based on what's present in the global counter byte. It'll then read the number of bytes to read from the key and value counters for each header from 0...n. The remainder of the binary data trailing the final header's value data will be the tile data with no specific buffer or separator. This is similar to how options work in DHCP packets.

The serializer will work in reverse, prepending bytes to the front of the tile data along with their counters until we place the global counter byte.