OryxProject / oryx

Oryx 2: Lambda architecture on Apache Spark, Apache Kafka for real-time large scale machine learning

Home Page:http://oryx.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

SpeedLayer read kafka offset

KnifeFly opened this issue · comments

Hi srowen, i am starter to this project. i checkout the code from master, and found file SpeedLayer.java update th kafka offset to the path /consumers in zk:

kafkaDStream.foreachRDD(new UpdateOffsetsFn<>(getGroupID(), getInputTopicLockMaster()));

The master branch use kafka auto commit offset character to save the offset to kafka. So there is no need to use UpdateOffsetsFn to update offset to zk in the end of spark task?

This should be on the mailing list. https://groups.google.com/a/cloudera.org/forum/#!forum/oryx-user
Which auto commit are you talking about?