rsilvestre / kafka-streams

equivalent to kafka-streams :octopus: for nodejs :sparkles::turtle::rocket::sparkles:

node-kafka-streams

Disclaimer: We are working on a new release that treats Keys and Values of Kafka Messages just as the official Java Version does it. Please be patient. It will also allow to define partitions and keys during pipes to output topics.

npm install --save kafka-streams

const {KafkaStreams} = require("kafka-streams");

const config = require("./config.json");
const factory = new KafkaStreams(config);

const kstream = factory.getKStream("input-topic");
const ktable = factory.getKTable(..);

kstream.merge(ktable).filter(..).map(..).reduce(..).to("output-topic");

API Overview

• Quick Start
• API Info
• Documentation
• Operator descriptions
• Examples
• Native Client | SSL, SASL, Kerberos

You might also like

• node-kafka-connect
• node-schema-registry
• node-kafka-rest-ui

README Overview

• Prerequisites
• Aim of this Library
• Description
• Port Progress Overview
• Operator Implementations
• Additional Operators
• Stream Action Implementations
• Join Operators Status
• Window Operations
• FAQ - More

Prerequisites

• kafka broker should be version >= 0.9.x, suggested: >= 0.10.x
• nodejs should be version >= 6.10, suggested: >= 8.6.x

Aim of this Library

• this is not a 1:1 port of the official JAVA kafka-streams
• the goal of this project is to give at least the same options to a nodejs developer that kafka-streams provides for JVM developers
• stream-state processing, table representation, joins, aggregate etc. I am aiming for the easiest api access possible checkout the word count example

Description

kafka-streams 🐙 equivalent for nodejs ✨🐢🚀✨ build on super fast 🔥 observables using most.js 🤘

ships with sinek 🙏 for backpressure

comes with js and native Kafka client, for more performance and SSL, SASL and Kerberos features

the lib also comes with a few window operations that are more similar to Apache Flink, yet they still feel natural in this api :squirrel:

overwriteable local-storage solution allows for any kind of datastore e.g. RocksDB, Redis, Postgres..

async (Promises) and sync stream operators e.g. stream$.map() or stream$.asyncMap()

super easy API

the lib is based on sinek, which is based on kafka-node's ConsumerGroups

Port Progress Overview

• core structure
• KStream base - stream as a changelog
• KTable base - stream as a database
• KStream & KTable cloning
• complex stream join structure
• advanced joins see
• windows (for joins) see
• flink like window operations
• word-count example
• more examples
• local-storage for etl actions
• local-storage factory (one per action)
• KStorage example for any DB that supports atomic actions
• backing-up local-storage via kafka
• kafka client implementation
• KTable replay to Kafka (produce)
• stream for topic message production only
• sinek implementation
• backpressure mode for KafkaClient
• auto-json payloads (read-map/write-map)
• auto producer partition and keyed-message handling
• documentation
• API description
• higher join & combine examples
• embed native client librdkafka for more performance
• SSL
• SASL
• Kerberos

Operator Implementations

• map
• asyncMap
• constant
• scan
• timestamp
• tap
• filter
• skipRepeats
• skipRepeatsWith
• slice
• take
• skip
• takeWhile
• skipWhile
• until
• since
• reduce
• chainReduce
• forEach (observe)
• chainForEach
• drain
• _zip
• _merge
• _join
• _combine
• _sample
• throttle
• debounce
• delay
• multicast
• A description of the operators can be found here
• Missing an operator? Feel free to open an issue 👮

Additional Operators

• mapStringToArray
• mapArrayToKV
• mapStringToKV
• mapParse
• mapStringify
• atThroughput
• mapWrapKafkaPayload
• mapToFormat
• mapFromFormat
• Want more? Feel free to open an issue 👮

Stream Action Implementations

• countByKey
• sumByKey
• min
• max
• Want more? Feel free to open an issue 👮

Join Operations

Operation description

KStream Status

• merge
• outerJoin
• innerJoin
• leftJoin

KTable Status

• merge
• outerJoin
• innerJoin
• leftJoin

KTable <-> KStream Status

• merge
• outerJoin
• innerJoin
• leftJoin

Window Operations

KStream

• window

More

Can I use this library yet?

Yes.

Are we ready for production yet?

Probably, yes. 😄

Even More

Forks or Stars give motivation