dashbitco / broadway_kafka

A Broadway connector for Kafka

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Allow to force consume the topic from the beginning or the end

danmarcab opened this issue · comments

The use case is always consuming the topic from beginning (to read the whole topic) or from the end (you only care about new messages), regardless of what the committed offsets were. See more info in #119

To support both cases we can add a config option begin_offset with the values :assigned and :reset and the following behaviour.

offset_reset_policy begin_offset outcome
:earliest :reset always read from start
:latest :reset always read from end
:earliest :assigned read from committed offset, or from start if not defined or expired (current behaviour)
:latest :assigned read from committed offset, or from end if not defined or expired (current behaviour)