lengthprefixed implements a Reader and Writer for reading, and writing, data in length-prefixed frames.
This is useful for efficiently streaming, or storing, large quantities of serialized data structures together when they are not self-delimiting, e.g. protocol buffer messages.
A length-prefixed frame adds 9-18 bytes of overhead to user-provided data:
a varint representing length of user-provided data, and then an 8-byte
(uint64
) checksum generated by the xxhash algorithm.
Each frame contains three parts: length (4 bytes), chunk data (length
bytes),
and checksum (8 bytes).
0 1-10 N N+8
+--------+================+----------+
| length | frame data | checksum |
+--------+================+----------+
The lengthprefixed API only requires that the user interact with the frame data. The length and checksum components are hidden from the user.
package main
import (
"bytes"
"fmt"
"github.com/dpirotte/lengthprefixed"
)
func main() {
var buf bytes.Buffer
w := lengthprefixed.NewWriter(&buf)
w.Write([]byte("This string would more likely be a protobuf message."))
r := lengthprefixed.NewReader(&buf)
b, err := r.ReadFrame()
if err != nil {
panic(err)
}
fmt.Println(string(b))
}