klarna-incubator / kflow

KFlow

Bullet-proof data stream processing framework

KFlow is an Erlang dataflow DSL built with Kafka in mind. Services implemented using Kflow have the following properties:

1. Stateless. Service can be restarted at any moment without risk of losing data
2. Fault-tolerant and self-healing. Any crashes in the service can be fixed, and the service will automatically replay the data that caused crash, once the fix is deployed. This enables fearless A/B testing and canary deployments
3. Scalable.
4. Automated parallelization. Each transformation runs in parallel
5. Automated backpressure.

This is achieved by rather sophisticated offset tracking logic built in KFlow behaviors, that makes sure that consumer offsets are committed to Kafka only when an input message is fully processed.

Another feature of KFlow is "virtual partition" that allows to split Kafka partitions into however many substream that are processed independently.

Usage example

KFlow is configured using a special Erlang module named kflow_config.erl. This module must export pipes/0 function returning a list of workflows. For example:

-module(kflow_config).

-export([pipes/0]).

pipes() -> [example_workflow()].

example_workflow() ->
  %% Define a "pipe" (much like Unix pipe):
  PipeSpec = [ %% Parse messages:
               {map, fun(_Offset, #{key => KafkaKey, value => JSON} ->
                         (jsone:decode(JSON)) #{key => KafkaKey}
                     end}
               %% Create a "virtual partition" for each unique key:
             , {demux, fun(_Offset, #{key := Key}) ->
                           Key
                       end}
               %% Collect messages into chunks of 100 for faster processing:
             , {aggregate, kflow_buffer, #{max_messages => 100}}
               %% Dump chunks into Postgres:
             , {map, kflow_postgres, #{ database => #{ host => "localhost"
                                                     , ...
                                                     }
                                      , table  => "my_table"
                                      , fields => [key, foo, bar, baz]
                                      , keys   => [key]
                                      }}
             ],
  kflow:mk_kafka_workflow(?FUNCTION_NAME, PipeSpec,
                          #{ kafka_topic => <<"example_topic">>
                           , group_id    => <<"example_consumer_group_id">>
                           }).

For more examples and usage, please refer to the Docs.

Development setup

KFlow requires OTP21 or later and rebar3 present in the PATH. Build by running

make

How to contribute

See our guide on contributing.

Release History

See our changelog.

License

Copyright © 2020 Klarna Bank AB

For license details, see the LICENSE file in the root of this project.