pipelinedb / pipeline_kafka

PipelineDB extension for Kafka support

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Partition offsets ignored when using consume_begin()

nielsdraaisma opened this issue · comments

I have two consumers configured in pipelinedb and when starting both using consume_begin() it outputs

LOG:  [pipeline_kafka] xxx <- xxx (PID 2316): consuming partition 8 from offset -1
LOG:  [pipeline_kafka] xxx <- xxx (PID 2316): consuming partition 2 from offset -1
...
LOG:  [pipeline_kafka] yyy <- yyy (PID 2317): consuming partition 8 from offset -1
LOG:  [pipeline_kafka] yyy <- yyy (PID 2317): consuming partition 11 from offset -1
...

this output is different from when i start the consumers individually using consume_begin('xxx', 'xxx', batchsize := 100)

LOG:  [pipeline_kafka] xxx <- xxx (PID 2360): consuming partition 8 from offset 1121144
LOG:  [pipeline_kafka] xxx <- xxx (PID 2360): consuming partition 11 from offset 1121012
...
LOG:  [pipeline_kafka] yyy <- yyy (PID 2375): consuming partition 8 from offset 2132108
LOG:  [pipeline_kafka] yyy <- yyy (PID 2375): consuming partition 2 from offset 2132065