is there a way to emulate the `ktables` functionality of kafka?
zcaudate opened this issue · comments
In a kafka streams app, the ktable keeps underlying state about the series coming in, acting like an accumulator based on the input key on the stream: https://docs.confluent.io/current/streams/concepts.html
I'm wondering if it's possible to do this type of transform. The simplest example I can is that of a sequence of accumulated min values:
input: [7 6 5 8 9 10 4 2 4]
output: [7 6 5 5 5 5 4 2 2]
or something like a gauge for metrics that counts a particular type of data that gets passed
input: [a b c a b d d a] ;; counter for b
output: [0 1 1 1 2 2 2 2]
another transform that the ktables does is to turn a reduced value back into a stream of deltas though it's harder to do with stateless reductions.
x/reductions is almost what you want except it will output one extra item.
Try something like (comp (x/reductions rfs/min) (drop 1)) for your first example and (comp (map #(case % b 1 0)) (x/reductions +) (drop 1)) for the second.
user=> (into [] (comp (x/reductions rfs/min) (drop 1)) [7 6 5 8 9 10 4 2 4])
[7 6 5 5 5 5 4 2 2]
user=> (into [] (comp (map #(case % b 1 0)) (x/reductions +) (drop 1)) '[a b c a b d d a])
[0 1 1 1 2 2 2 2]oh nice. that's very cool.