java.lang.NoClassDefFoundError: org/apache/kafka/server/authorizer/Authorizer when doing EmbeddedKafka.start()

[srini-am]

When I am trying to write unit test for Spark streaming scala with kafka, I am getting following error, Can you please suggest right combination of dependencies or how do we resolve.

A needed class was not found. This could be due to an error in your runpath. Missing class: org/apache/kafka/server/authorizer/Authorizer
java.lang.NoClassDefFoundError: org/apache/kafka/server/authorizer/Authorizer
at kafka.server.KafkaConfig$.(KafkaConfig.scala:520)
at kafka.server.KafkaConfig$.(KafkaConfig.scala)
at net.manub.embeddedkafka.ops.KafkaOps$class.startKafka(kafkaOps.scala:36)

Kafka dependencies in pom.xml:

    <dependency>
        <groupId>com.github.benfradet</groupId>
        <artifactId>spark-kafka-0-10-writer_2.11</artifactId>
        <version>0.2.0</version>
    </dependency>

    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql-kafka-0-10_2.11</artifactId>
        <version>2.4.4</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.github.embeddedkafka/embedded-kafka-streams -->
    <dependency>
        <groupId>io.github.embeddedkafka</groupId>
        <artifactId>embedded-kafka-streams_2.11</artifactId>
        <version>2.4.1.1</version>
    </dependency>

Thank You,
Srinivas

[Francesco Pellegrini]

Hi @srini-am, although I'm not familiar with Spark transitive dependencies, I'm aware of the important limitation mentioned in the README of this project.

Please check your Scala version and make sure it matches Kafka's.