dpirotte / go-lengthprefixed

Read and write streams of irregularly sized "chunks" of bytes

lengthprefixed

lengthprefixed implements a Reader and Writer for reading, and writing, data in length-prefixed frames.

This is useful for efficiently streaming, or storing, large quantities of serialized data structures together when they are not self-delimiting, e.g. protocol buffer messages.

A length-prefixed frame adds 9-18 bytes of overhead to user-provided data: a varint representing length of user-provided data, and then an 8-byte (uint64) checksum generated by the xxhash algorithm.

Frame Structure

Each frame contains three parts: length (4 bytes), chunk data (length bytes), and checksum (8 bytes).

0       1-10              N         N+8
+--------+================+----------+
| length |   frame data   | checksum |
+--------+================+----------+

The lengthprefixed API only requires that the user interact with the frame data. The length and checksum components are hidden from the user.

Usage

package main

import (
  "bytes"
  "fmt"

  "github.com/dpirotte/lengthprefixed"
)

func main() {
  var buf bytes.Buffer
  w := lengthprefixed.NewWriter(&buf)

  w.Write([]byte("This string would more likely be a protobuf message."))

  r := lengthprefixed.NewReader(&buf)
  b, err := r.ReadFrame()
  if err != nil {
    panic(err)
  }

  fmt.Println(string(b))
}