awslabs / flowgger

A fast data collector in Rust

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Flowgger

CI Build License: BSD2

New major version: 0.3.0 (2022-03-14)

Breaking Changes


Flowgger is a fast, simple and lightweight data collector written in Rust.

It reads log entries over a given protocol, extracts them, decodes them using a given format, re-encodes them into a different format, and asynchronously pushes the result into a remote data store.

Flowgger is designed to be:

  • Paranoid: it carefully validates input data to prevent injection of malformed/incomplete records down the chain.
  • Safe: written in Rust, without any unsafe code.
  • Fast: even though messages are systematically parsed and validated, Flowgger is orders of magnitude faster than Logstash and Fluentd.
  • Standalone: it comes as a single executable file, and doesn't require a JVM.

Flowgger supports common input types: stdin, UDP, TCP, TLS and Redis, as well as multiple input formats: JSON (GELF), LTSV, Cap'n Proto and RFC5424. Normalized messages can be sent to Kafka, Graylog, to downstream Flowgger servers, or to other log collectors for further processing.

About

A fast data collector in Rust

License:BSD 2-Clause "Simplified" License


Languages

Language:Rust 99.6%Language:Dockerfile 0.2%Language:Cap'n Proto 0.2%Language:Shell 0.0%