katallaxie / streams

🏄‍♀️ A teeny-tiny package to create stream processing workloads.

🏄‍♀️ Streams

A teeny-tiny package to create stream processing workloads. It is intended to be used with Apache Kafka.

Getting Started

There are only a few packages that help Gophers to create stream processing workloads. This package is one of them. It is intended to be used with Apache Kafka.

Supported:

• Apache Kafka
• nats.io (experimental)

go get github.com/katallaxie/streams

It features a channel based API to consume messages from a Kafka topic and a channel based API to produce messages to a Kafka topic. It assumes the use of a consumer group for the consumption of messages.

There is a source which feeds messages from a source into a stream. There is a sink which writes processed messages into a data sink.

When using a sink in the procesing the commit mode can be set to CommitManual which means that the sink will not commit the offset of the consumed message. This is useful when the message is processed in a stream and the offset should only be committed after the message has been processed.

The package connects a source with a sink via small functional operatios.

• Branch
• Do
• FanOut
• Filter
• Log
• Map
• Merge
• Table (experimental) Stores the message in a table. The table can be queried via a View.

There is support for Prometheus metrics.

The view package provides the ability to create views with data from the stream processing workload.

Docs

You can find the documentation hosted on godoc.org.

License

Apache 2.0