Go VRL

Experimental go bindings for Vector Remap Language

Vector Remap Language (VRL) is an expression-oriented language designed for transforming observability data (logs and metrics) in a safe and performant manner. It features a simple syntax and a rich set of built-in functions tailored specifically to observability use cases.

Versions

There are two major versions of this module and consumers must choose which is a better fit for their use case.

They aim to support as similar an interface as possible, with the key distinction being how VRL programs are executed.

V5 uses cgo to interface with a custom library built from VRL. This has better performance with the main downside being that it relies on cgo, which some applications may not care for.
V10 uses wasm to execute VRL. It performs worse, on the order of 2-3 times slower, however VRL is quite efficient so this still offers relatively good absolute performance.

Usage

Feature Support

	V5	V10
Compiling a VRL Program	✅	✅
Running a VRL Program	✅	✅
VRL Runtime "Basic"* API	✅	✅
Environment Kinds	'Byte' and 'Object'	❌
Secrets	❌	❌
Metadata	❌	❌
Timezones	❌	❌
Requires CGO	✅	❌
Full VRL stdlib support	✅	❌*

* "Basic" API currently means:

compile
resolve (run) the compiled program
clear
is_empty

* WASM supports almost most of VRL's stdlib functions, the unsupported ones can be found with this GH issues search

Building and importing

Not quite ready yet. It's difficult to distribute a go module that depends on an external build system, we have some ideas though.

To use this repo as-is, its required to manually compile the rust dependency. For V5: cd v5; cargo build --release; cd example/; go run . For V10: cd v10; cargo build --target wasm32-wasi --release; cd example/; go run .

Examples

V5

program, err := govrl.CompileWithExternal(`replace(., "go", "rust")`, govrl.GetExternalEnv(govrl.Bytes, govrl.Bytes))
if err != nil {
    log.Panicln(err)
}

runtime := govrl.NewRuntime()
res, err := runtime.Resolve(program, "hello go")
if err != nil {
    log.Panicln(err)
}

fmt.Println(res)

$ go run .
hello rust

see ./v5/example/main.go for more examples

V10

package main

import (
	"context"
	"fmt"
	"log"

	govrl "github.com/gh123man/go-vrl/v10"
)

func main() {
	simpleDefault()
}

func simpleDefault() {
	ctx := context.Background()
	wasmInterface := govrl.NewWasmInterface(ctx)
	program, err := wasmInterface.Compile(`
	. = parse_json!(string!(.))
	del(.foo)

	.timestamp = now()

	http_status_code = parse_int!(.http_status)
	del(.http_status)

	if http_status_code >= 200 && http_status_code <= 299 {
		.status = "success"
	} else {
		.status = "error"
	}
	.
	`)

	if err != nil {
		log.Panicln(err)
		return
	}

	runtime, err := wasmInterface.NewRuntime()
	if err != nil {
		log.Panicln(err)
	}

	res, err := runtime.Resolve(program, `{
		"message": "Hello VRL",
		"foo": "delete me",
		"http_status": "200"
	}
	`)
	if err != nil {
		fmt.Println(err)
		return
	}

	fmt.Println(res)
	runtime.Clear()
}

$ go run .
{ "message": "Hello VRL", "status": "success", "timestamp": t'2022-01-01T00:00:00Z' }

see ./v10/example/main.go for more examples

gh123man / go-vrl