Unable to receive data

Question

Unable to receive data

sorabh89 opened this issue 9 years ago · comments

Hi Dibbhatt,

A am facing some issue after changing one of the zookeeper nodes. I've changed all the settings related to zookeeper ip's. But getting the following:

ERROR ReceiverTracker: Deregistered receiver for stream 1: Error starting receiver 1 - java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:73)
at consumer.kafka.ZkCoordinator.(ZkCoordinator.java:64)
at consumer.kafka.KafkaConsumer.open(KafkaConsumer.java:61)
at consumer.kafka.client.KafkaRangeReceiver.start(KafkaRangeReceiver.java:73)
at consumer.kafka.client.KafkaRangeReceiver.onStart(KafkaRangeReceiver.java:58)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:308)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:300)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:117)
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:62)
... 16 more
Caused by: java.lang.IllegalStateException: instance must be started before calling this method
at org.spark-project.guava.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:360)
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:110)
... 17 more

15/09/15 12:56:45 ERROR ReceiverTracker: Deregistered receiver for stream 1: Error starting receiver 1 - java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:73)
at consumer.kafka.ZkCoordinator.(ZkCoordinator.java:64)
at consumer.kafka.KafkaConsumer.open(KafkaConsumer.java:61)
at consumer.kafka.client.KafkaRangeReceiver.start(KafkaRangeReceiver.java:73)
at consumer.kafka.client.KafkaRangeReceiver.onStart(KafkaRangeReceiver.java:58)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:308)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:300)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:117)
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:62)
... 16 more
Caused by: java.lang.IllegalStateException: instance must be started before calling this method
at org.spark-project.guava.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:360)
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:110)
... 17 more

entered forEachRDD
[Stage 2:> (0 + 4) / 4][Stage 3:> (0 + 0) / 14]
15/09/15 12:57:24 ERROR ReceiverTracker: Deregistered receiver for stream 0: Error starting receiver 0 - java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:73)
at consumer.kafka.ZkCoordinator.(ZkCoordinator.java:64)
at consumer.kafka.KafkaConsumer.open(KafkaConsumer.java:61)
at consumer.kafka.client.KafkaRangeReceiver.start(KafkaRangeReceiver.java:73)
at consumer.kafka.client.KafkaRangeReceiver.onStart(KafkaRangeReceiver.java:58)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver(ReceiverSupervisor.scala:125)
at org.apache.spark.streaming.receiver.ReceiverSupervisor.start(ReceiverSupervisor.scala:109)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:308)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:300)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1767)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63)
at org.apache.spark.scheduler.Task.run(Task.scala:70)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.IllegalStateException: instance must be started before calling this method
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:117)
at consumer.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:62)
... 16 more
Caused by: java.lang.IllegalStateException: instance must be started before calling this method
at org.spark-project.guava.base.Preconditions.checkState(Preconditions.java:149)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getData(CuratorFrameworkImpl.java:360)
at consumer.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:110)
... 17 more

Regards,
Sorabh

Dibyendu Bhattacharya commented 9 years ago

Cool

Dibyendu Bhattacharya · Answer 1 · Tue Sep 15 2015 17:37:51 GMT+0800 (China Standard Time)

Did you change the ZK settings while Streaming job is still running ?

Sorabh Kumar · Answer 2 · Tue Sep 15 2015 17:39:02 GMT+0800 (China Standard Time)

No, I stopped the streaming and then changes the settings.

Dibyendu Bhattacharya · Answer 3 · Tue Sep 15 2015 17:43:19 GMT+0800 (China Standard Time)

There are two ZK Configuration .

The ZK which Kafka uses .. This to get connected to kafka partitions. This is the zookeeper.hosts and zookeeper.ports property.
The ZK where offset are committed. This is zookeeper.consumer.connection property.

Both can point to same ZK cluster. But in some cases people may prefer not to touch the ZK which control Kafka .

Which ZK settings you changed ? if you haven't touch the Kafka cluster ZK settings, then you do not need to change 1 . If you want to commit offset to different ZK cluster, you can change 2.

Dibyendu Bhattacharya · Answer 4 · Tue Sep 15 2015 17:45:01 GMT+0800 (China Standard Time)

It may be possible your ZK is not UP . Do you able to connect to Kafka Cluster from Console using the changed ZK settings ?

Dibyendu Bhattacharya · Answer 5 · Tue Sep 15 2015 17:46:00 GMT+0800 (China Standard Time)

bin/kafka-console-consumer.sh --zookeeper 10.252.1.136:2181 --topic mytopic --from-beginning

Try this from Console and see if you are getting kafka messages ..

Sorabh Kumar · Answer 6 · Tue Sep 15 2015 17:46:18 GMT+0800 (China Standard Time)

I've changed it at both the places, and yes it is working from console.

Sorabh Kumar · Answer 7 · Tue Sep 15 2015 17:47:38 GMT+0800 (China Standard Time)

I am getting some error related to brokerInfo. And since while connecting from console, I don't provide broker info, its working from console.

Dibyendu Bhattacharya · Answer 8 · Tue Sep 15 2015 17:47:46 GMT+0800 (China Standard Time)

Ok, then goto zk..

and remove the earlier consumer path ..

or use new kafka.consumer.id

Sorabh Kumar · Answer 9 · Tue Sep 15 2015 17:48:54 GMT+0800 (China Standard Time)

I've tried using new consumer id also, but getting the same error.

Dibyendu Bhattacharya · Answer 10 · Tue Sep 15 2015 17:49:21 GMT+0800 (China Standard Time)

It may be also possible, you Spark cluster where Executors are running , not able to connect to new ZK IP. Check the etc/passwd file

Sorabh Kumar · Answer 11 · Tue Sep 15 2015 17:50:20 GMT+0800 (China Standard Time)

Actually when I start I get this error but I also get some data. and then after 2-3 batches it shows 0 records.

Dibyendu Bhattacharya · Answer 12 · Tue Sep 15 2015 17:55:51 GMT+0800 (China Standard Time)

So the error you mentioned is not there now ?

Dibyendu Bhattacharya · Answer 13 · Tue Sep 15 2015 17:58:31 GMT+0800 (China Standard Time)

Which version of ZK you installed ? Is it different than earlier version ? There could be also possibility that the Curator Framework not able to connect to ZK properly

Sorabh Kumar · Answer 14 · Tue Sep 15 2015 17:58:33 GMT+0800 (China Standard Time)

It is there, but after printing the error the program continues.

Sorabh Kumar · Answer 15 · Tue Sep 15 2015 17:58:56 GMT+0800 (China Standard Time)

No the version of ZK is same.

Dibyendu Bhattacharya · Answer 16 · Tue Sep 15 2015 17:59:36 GMT+0800 (China Standard Time)

in Spark UI do you see Receivers are running ?

Sorabh Kumar · Answer 17 · Tue Sep 15 2015 18:00:31 GMT+0800 (China Standard Time)

You mean, executors?

Dibyendu Bhattacharya · Answer 18 · Tue Sep 15 2015 18:03:53 GMT+0800 (China Standard Time)

yes. It says ERROR ReceiverTracker: Deregistered receiver . Not sure is your Receivers are still running

Sorabh Kumar · Answer 19 · Tue Sep 15 2015 18:15:30 GMT+0800 (China Standard Time)

They are running, earlier it was working fine but after changing the zookeeper ip, this error started showing up.

Sorabh Kumar · Answer 20 · Tue Sep 15 2015 20:13:16 GMT+0800 (China Standard Time)

Got resolved after restarting Kafka and zookeeper.

Thanks,