Problems getting MSK IAM Jar to co-operate with julie

[Edward Muller]

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

1. Download julie-ops.jar & aws-msk-iam-auth jar
2. run java -cp ~/jars/aws-msk-iam-auth-1.1.4-all.jar -jar ~/jars/julie-ops.jar ....
3. get error

org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
	at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:757)
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:503)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:496)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
	at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:233)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:65)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
	at com.purbon.kafka.topology.api.adminclient.TopologyBuilderAdminClientBuilder.build(TopologyBuilderAdminClientBuilder.java:28)
	at com.purbon.kafka.topology.JulieOps.build(JulieOps.java:80)
	at com.purbon.kafka.topology.CommandLineInterface.processTopology(CommandLineInterface.java:211)
	at com.purbon.kafka.topology.CommandLineInterface.run(CommandLineInterface.java:161)
	at com.purbon.kafka.topology.CommandLineInterface.main(CommandLineInterface.java:147)

Expected behavior
I expected adding the aws-msk-iam-auth to the classpath to work.

Essentially I expected this to work java -cp ~/jars/aws-msk-iam-auth-1.1.4-all.jar -jar ~/jars/julie-ops.jar ....

Screenshots
If applicable, add screenshots to help explain your problem.

Runtime (please complete the following information):

• OS: macOS 12.3.1
• JVM version: openjdk 18.0.1 2022-04-19

Additional context

I pulled apart the julie-ops.jar file and found some stuff already in software/amazon/ and I think that is confusing the class loader. I am no java expert though. I checked the repo out and applied this patch and the new jar works without having to add the aws-msk-iam-auth jar to the classpath (because it's part of the julie-ops jar)

diff --git a/pom.xml b/pom.xml
index 6f45a98..8f50be1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -402,6 +402,7 @@
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-source-plugin</artifactId>
+        <version>3.2.1</version>
         <executions>
           <execution>
             <id>attach-sources</id>
@@ -519,7 +520,8 @@
     <hamcrest.version>2.2</hamcrest.version>
     <assertj.version>3.15.0</assertj.version>
     <jinjava.version>2.5.4</jinjava.version>
-    <aws.java.sdk.version>2.16.31</aws.java.sdk.version>
+    <aws.java.sdk.version>2.17.192</aws.java.sdk.version>
+    <aws.msk.iam.auth.version>1.1.4</aws.msk.iam.auth.version>
     <gcp.java.sdk.version>19.2.1</gcp.java.sdk.version>
     <ksqldb.version>0.17.0</ksqldb.version>
     <typesafe.version>1.4.0</typesafe.version>
@@ -654,6 +656,11 @@
       <groupId>software.amazon.awssdk</groupId>
       <artifactId>s3</artifactId>
     </dependency>
+    <dependency>
+      <groupId>software.amazon.msk</groupId>
+      <artifactId>aws-msk-iam-auth</artifactId>
+      <version>${aws.msk.iam.auth.version}</version>
+    </dependency>
     <dependency>
       <groupId>com.google.cloud</groupId>
       <artifactId>google-cloud-storage</artifactId>
@@ -724,6 +731,13 @@
         <type>pom</type>
         <scope>import</scope>
       </dependency>
+      <dependency>
+        <groupId>software.amazon.msk</groupId>
+        <artifactId>aws-msk-iam-auth</artifactId>
+        <version>${aws.msk.iam.auth.version}</version>
+        <type>pom</type>
+        <scope>import</scope>
+      </dependency>
       <dependency>
         <groupId>com.google.cloud</groupId>
         <artifactId>libraries-bom</artifactId>

[Pere Urbón]

Moin,
the error you're getting here is with the configuration, do you mind sharing your config with us? this would certainly help to evaluate what might be happening.

btw, re MSK custom integration, some context look at #458

[Edward Muller]

ssl.truststore.location=/usr/local/opt/openjdk/libexec/openjdk.jdk/Contents/Home/lib/security/cacerts
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

This is the same config that I am using post bundling and with the standard kafka tools.

[Khanh Phan (Chris)]

The following configs worked in my case when access MSK with IAM auth
client.properties

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
topology.topic.prefix.format={{topic}}
topology.project.prefix.format=

app.sh

set -e
JULIE_OPS_HOME=/usr/local/julie-ops
JULIE_OPS_JAR=$JULIE_OPS_HOME/bin/julie-ops.jar
java -cp aws-msk-iam-auth.jar:$JULIE_OPS_JAR com.purbon.kafka.topology.CommandLineInterface --topology descriptor.yaml --clientConfig client.properties --brokers $MSK_BBROKERS 
exec "$@"