Giters

pinterest / DoctorK

DoctorK is a service for Kafka cluster auto healing and workload balancing

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Concurrency issue

BrianGallew opened this issue · comments

commented

When I do a largish number of HTTP GET requests against /servlet/clusterinfo on a largish DoctorKafka config, I occasionally see the following error logged:

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1824) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:1808) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.apache.kafka.clients.consumer.KafkaConsumer.listTopics(KafkaConsumer.java:1524) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at com.pinterest.doctorkafka.KafkaClusterManager.toJson(KafkaClusterManager.java:146) ~[doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at com.pinterest.doctorkafka.servlet.ClusterInfoServlet.renderJSON(ClusterInfoServlet.java:43) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at com.pinterest.doctorkafka.servlet.DoctorKafkaServlet.doGet(DoctorKafkaServlet.java:130) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:867) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at io.dropwizard.servlets.ThreadNameFilter.doFilter(ThreadNameFilter.java:35) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.handle(AllowedMethodsFilter.java:45) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at io.dropwizard.jersey.filter.AllowedMethodsFilter.doFilter(AllowedMethodsFilter.java:39) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at io.dropwizard.jetty.RoutingHandler.handle(RoutingHandler.java:52) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:174) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.Server.handle(Server.java:502) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) [doctorkafka-0.2.4.3-jar-with-dependencies.jar:?]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
commented

No longer using DoctorKafka