SpeedLayer read kafka offset
KnifeFly opened this issue · comments
KnifeFly commented
Hi srowen, i am starter to this project. i checkout the code from master, and found file SpeedLayer.java update th kafka offset to the path /consumers in zk:
kafkaDStream.foreachRDD(new UpdateOffsetsFn<>(getGroupID(), getInputTopicLockMaster()));
The master branch use kafka auto commit offset character to save the offset to kafka. So there is no need to use UpdateOffsetsFn to update offset to zk in the end of spark task?
Sean Owen commented
This should be on the mailing list. https://groups.google.com/a/cloudera.org/forum/#!forum/oryx-user
Which auto commit are you talking about?