slogproto

Warning This is an experimental module and is subject to change.

Go slog.Handler using Protocol Buffers. This can reduce the size of log messages when saving them to disk or sending them over the network, and can reduce the amount of time spent marshaling and unmarshaling log messages at the cost of human readability. To enable interopability with other tools, the slp CLI can read protobuf encoded slog.Records from STDIN (or a file) and output them as JSON to STDOUT. Logs can be filtered using CEL expressions.

Installation

For use with Go programs:

$ go get -u -v github.com/picatz/slogproto@latest

To use the slp CLI:

$ go install -v github.com/picatz/slogproto/cmd/slp@latest

Usage

package main

import (
	"os"

	"github.com/picatz/slogproto"
	"golang.org/x/exp/slog"
)

func main() {
	logger := slog.New(slogproto.NewHander(os.Stdout))

	logger.Info("example", slog.Int("something", 1))
}

$ go run main.go | slp
{"time":"2023-08-01T03:12:11.272826Z","level":"INFO","msg":"example","something":1}

$ slp output.log
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
{"time":"..","level":"...","msg":"...", ... }
...

Note Input to slp can be from STDIN or a file.

Filtering

The filter flag can be used to filter logs using a given expression. The expression is evaluated against the slog.Record and must return a boolean value. For each log record that the expression evaluates as true will be output to STDOUT as JSON.

attrs is a map of all the attributes in the log record, not including the message, level, or time.
```
 attrs.something == 1
```
```
 has(attrs.something) && attrs.something == 1
```
Important Invalid access to an attribute will cause the filter to fail at evaluation time. Wrap the expression in a has() function to check if the attribute exists before accessing it.
msg is the message in the log record.
level is the level in the log record.
time is the timestamp in the log record.

$ slp --filter='has(attrs.something)' output.log
{"time":"2023-08-11T00:06:00.474782Z","level":"INFO","msg":"example","something":1}
$  --filter='msg == "this is a test"' test.log
{"time":"2023-08-11T00:06:00.474033Z","level":"INFO","msg":"this is a test","test":{"test2":"1","test3":1,"test1":1}}

File Format

The file format is a series of delimited Protocol Buffer messages. Each message is prefixed with a 32-bit unsigned integer representing the size of the message. The message itself is a protobuf encoded slog.Record.

╭────────────────────────────────────────────────────────────╮
│  Message Size  │  Protocol Buffer Message  │  ...  │  EOF  │
╰────────────────────────────────────────────────────────────╯

Comparisons to Other Formats

Using the following record written 1024 times:

{
	"time": $timestamp,
	"level": "INFO",
	"msg": "hello world",
	"i": $n
}

Format	GZIP (KB)	Snappy (KB)	Zstandard (KB)	Uncompressed (KB)
Protobuf	5.48	11.17	3.58	41.88
JSON	5.79	9.59	5.04	86.81
Text	2.93	7.66	1.31	69.92

picatz / slogproto