Not able to start kafkastats
jmarkan opened this issue · comments
We're getting this error when starting kafkastats:
Unrecognized option: -jmxport=9999
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Could you share more details on how you started kafkastats? We run kafkastat in production with command that is similar to the following, and it works fine.
java -server -cp /opt/kafkastats:/opt/kafkastats/*:/opt/kafkastats/lib/*
com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 9999 -topic brokerstats -zookeeper
test001:2181/ -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -
pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
we also have the following line included in kafka command line:
-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.rmi.port=9999
@yuyang08 Here is the command that I'm attempting:
java -server -cp /opt/doctorkafka/kafkastats:/opt/doctorkafka/kafkastats/*:/opt/doctorkafka/kafkastats/src/main/java/com/pinterest/doctorkafka/stats/* -Dlog4j.configuration=com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 1098 -topic brokerstats -zookeeper <ZK_host1>:2181,<ZK_host2>:2181,<ZK_host3>:2181 -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
I updated the jmx port from 9999 to 1098 (the one that kafka is actually using) but still getting the unrecognized error.
@jmarkan please remove -Dlog4j.configuration= from the command line, or add a space between -Dlog4j.configuration= and com.pinterest.doctorkafka.stats.KafkaStatsMain. In the README.md doc, I did not specify the proper log4j.configuration path. Will update the doc on that.
@yuyang08 I'm getting a different error this time when I added the space, like you suggested:
java -server -cp /opt/doctorkafka/kafkastats:/opt/doctorkafka/kafkastats/*:/opt/doctorkafka/kafkastats/src/main/java/com/pinterest/doctorkafka/stats/* -Dlog4j.configuration= com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 1098 -topic brokerstats -zookeeper <ZK-host1>:2181,<ZK-host2>:2181,<ZK-host3>:2181 -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
Error: Could not find or load main class com.pinterest.doctorkafka.stats.KafkaStatsMain
I found the solution @jmarkan. Up to today I was also facing the same issue.
You just need to give class path of target folders. And I think so that you are missing lib folder in class path.
you can make use of
java -server -cp /opt/doctorkafka/kafkastats/target:/opt/doctorkafka/kafkastats/target/*:/opt/doctorkafka/kafkastats/target/lib/* -Dlog4j.configuration= com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 1098 -topic brokerstats -zookeeper <ZK-host1>:2181,<ZK-host2>:2181,<ZK-host3>:2181 -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
And the most important thing is you need to keep space after -Dlog4j.configuration=.
I think so it might help you.
Awesome, thanks @abhiverma23 this helped. I got over the error, however I'm getting a new one now. Here is my command and corresponding output:
java -server -cp /opt/doctorkafka/kafkastats/target:/opt/doctorkafka/kafkastats/target/*:/opt/doctorkafka/kafkastats/target/lib/* -Dlog4j.configuration= com.pinterest.doctorkafka.stats.KafkaStatsMain -jmxport 1098 -topic brokerstats -zookeeper <ZK-host1>:2181,<ZK-host2>:2181,<ZK-host3>:2181 -tsdhostport localhost:18126 -ostrichport 2051 -uptimeinseconds 3600 -pollingintervalinseconds 60 -kafka_config /etc/kafka/server.properties
Output:
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/doctorkafka/kafkastats/target/kafkastats-0.1.0-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/doctorkafka/kafkastats/target/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] log4j:WARN No appenders could be found for logger (com.pinterest.doctorkafka.util.OstrichAdminService). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 16:38:52.808 [StatsReporter] ERROR com.pinterest.doctorkafka.stats.BrokerStatsRetriever - Failed to get ec2 metadata java.io.IOException: Cannot run program "ec2metadata": error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[?:1.8.0_141] at java.lang.Runtime.exec(Runtime.java:620) ~[?:1.8.0_141] at java.lang.Runtime.exec(Runtime.java:450) ~[?:1.8.0_141] at java.lang.Runtime.exec(Runtime.java:347) ~[?:1.8.0_141] at com.pinterest.doctorkafka.stats.BrokerStatsRetriever.setBrokerInstanceInfo(BrokerStatsRetriever.java:358) [kafkastats-0.1.0-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.stats.BrokerStatsRetriever.retrieveBrokerStats(BrokerStatsRetriever.java:519) [kafkastats-0.1.0-jar-with-dependencies.jar:?] at com.pinterest.doctorkafka.stats.BrokerStatsReporter.run(BrokerStatsReporter.java:57) [kafkastats-0.1.0-jar-with-dependencies.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_141] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_141] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_141] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141] Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) ~[?:1.8.0_141] at java.lang.UNIXProcess.<init>(UNIXProcess.java:247) ~[?:1.8.0_141] at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[?:1.8.0_141] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[?:1.8.0_141] ... 13 more Mar 09, 2018 4:38:53 PM com.twitter.ostrich.admin.BackgroundProcess start INFO: Starting LatchedStatsListener Mar 09, 2018 4:38:53 PM com.twitter.ostrich.admin.AdminHttpService start INFO: Admin HTTP interface started on port 2051.
It looks like it's not able to get the ec2 instance information.
You might want to check latest. I think a fix went out for this issue.
Thanks @BrianGallew. But we went ahead with using LinkedIn's "Cruise Control" as our preferred solution instead of DoctorKafka. Thanks anyways.