Partition offsets ignored when using consume_begin()
nielsdraaisma opened this issue · comments
Niels Draaisma commented
I have two consumers configured in pipelinedb and when starting both using consume_begin() it outputs
LOG: [pipeline_kafka] xxx <- xxx (PID 2316): consuming partition 8 from offset -1
LOG: [pipeline_kafka] xxx <- xxx (PID 2316): consuming partition 2 from offset -1
...
LOG: [pipeline_kafka] yyy <- yyy (PID 2317): consuming partition 8 from offset -1
LOG: [pipeline_kafka] yyy <- yyy (PID 2317): consuming partition 11 from offset -1
...
this output is different from when i start the consumers individually using consume_begin('xxx', 'xxx', batchsize := 100)
LOG: [pipeline_kafka] xxx <- xxx (PID 2360): consuming partition 8 from offset 1121144
LOG: [pipeline_kafka] xxx <- xxx (PID 2360): consuming partition 11 from offset 1121012
...
LOG: [pipeline_kafka] yyy <- yyy (PID 2375): consuming partition 8 from offset 2132108
LOG: [pipeline_kafka] yyy <- yyy (PID 2375): consuming partition 2 from offset 2132065