Confluent hub installation is missing guava

Question

Confluent hub installation is missing guava

mmajis opened this issue 3 years ago · comments

Hi!

Thanks for writing this config provider!

It looks like guava is a dependency for your connect-utils library but it's not included in the confluent hub installation for this config provider.

I installed the provider to cp-kafka-connect:6.2.0:

FROM confluentinc/cp-kafka-connect:6.2.0

RUN confluent-hub install --no-prompt confluentinc/kafka-connect-s3:10.0.0 && \
    confluent-hub install --no-prompt jcustenborder/kafka-config-provider-aws:0.1.1

The container wouldn't start until I added guava-30.1.1-jre.jar in /usr/share/confluent-hub-components/jcustenborder-kafka-config-provider-aws/lib and modified CUB_CLASSPATH to add all the jars there:

export CUB_CLASSPATH="$CUB_CLASSPATH:\"/usr/share/confluent-hub-components/jcustenborder-kafka-config-provider-aws/lib/*\""

I'm not sure if this is a cub issue only or would it fail without guava if it gets past the cub health check. To begin with, I got a ClassDefNotFoundError for the config provider class itself, so definitely cub is not ready to handle config providers installed via confluent-hub without additional classpath config.

Guava related stack trace:

===> Check if Kafka is healthy ...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/share/java/cp-base-new/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.SimpleLoggerFactory]
Exception in thread "main" java.lang.NoClassDefFoundError: com/google/common/base/Strings
at com.github.jcustenborder.kafka.connect.utils.config.ConfigKeyBuilder.build(ConfigKeyBuilder.java:61)
at com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProviderConfig.config(SecretsManagerConfigProviderConfig.java:75)
at com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProviderConfig.&lt;init&gt;(SecretsManagerConfigProviderConfig.java:53)
at com.github.jcustenborder.kafka.config.aws.SecretsManagerConfigProvider.configure(SecretsManagerConfigProvider.java:136)
at org.apache.kafka.common.config.AbstractConfig.instantiateConfigProviders(AbstractConfig.java:572)

It seems the cub classpath is an issue in general for config providers, see confluentinc/cp-docker-images#828.

Peter Cummuskey · Answer 1 · Tue Aug 02 2022 12:44:30 GMT+0800 (China Standard Time)

Can confirm that it's also an issue in MSK (which has isolated plugin classpaths).

Fuxin Hao · Answer 2 · Fri Sep 23 2022 15:48:26 GMT+0800 (China Standard Time)

is there any progress?

Spyros Angelopoulos · Answer 3 · Thu Oct 20 2022 21:42:46 GMT+0800 (China Standard Time)

Any news on how this can be fixed/bypassed? Same issue happening on AWS MSK.

Ethan Davis · Answer 4 · Tue Nov 22 2022 03:31:58 GMT+0800 (China Standard Time)

Yeah I'm having this same issue MSK as well.

savealive · Answer 5 · Tue Nov 29 2022 11:50:46 GMT+0800 (China Standard Time)

same issue on MSK

Colin Nieuwenhuizen · Answer 6 · Thu Dec 01 2022 11:01:11 GMT+0800 (China Standard Time)

To resolve the MSK Connect issue I downloaded the guava jar (guava-31.1-jre.jar) directly from here

When you create the custom plugin for MSK Connect, after extracting jcusten-border-kafka-config-provider-aws drop the guava jar in the lib folder, before creating the archive that you upload to S3.

Ethan Davis · Answer 7 · Fri Dec 02 2022 22:13:37 GMT+0800 (China Standard Time)

@cgn-ca thanks for the help that fixed it for me!

Anand Kothapeta · Answer 8 · Wed Dec 14 2022 02:30:05 GMT+0800 (China Standard Time)

I ran into the same issue using this with Debezium oracle cdc connector v2.x.. It works fine without guava jar with V1.9.7. Once I copied the guava.31.1-jre.jar V2.x works fine..

Jacob Ray Gilman · Answer 9 · Wed Feb 08 2023 05:21:45 GMT+0800 (China Standard Time)

To resolve the MSK Connect issue I downloaded the guava jar (guava-31.1-jre.jar) directly from here

When you create the custom plugin for MSK Connect, after extracting jcusten-border-kafka-config-provider-aws drop the guava jar in the lib folder, before creating the archive that you upload to S3.

Maybe I'm doing something not quite right here, but I grabbed the same file, and placed it at: debezium-connector-sqlserver-2.1.2/jcustenborder-kafka-config-provider-aws-0.1.2/lib/guava-31.1-jre.jar and then reuploaded to S3. Does this require re-creating the custom plugin?

The debezium-connector-sqlserver-2.1.2 folder ends up being what gets zipped. Is this similar to your directory structure?

Anand Kothapeta · Answer 10 · Wed Feb 08 2023 05:31:40 GMT+0800 (China Standard Time)

To resolve the MSK Connect issue I downloaded the guava jar (guava-31.1-jre.jar) directly from here
When you create the custom plugin for MSK Connect, after extracting jcusten-border-kafka-config-provider-aws drop the guava jar in the lib folder, before creating the archive that you upload to S3.

Maybe I'm doing something not quite right here, but I grabbed the same file, and placed it at: debezium-connector-sqlserver-2.1.2/jcustenborder-kafka-config-provider-aws-0.1.2/lib/guava-31.1-jre.jar and then reuploaded to S3. Does this require re-creating the custom plugin?

The debezium-connector-sqlserver-2.1.2 folder ends up being what gets zipped. Is this similar to your directory structure?

The directory structure is different for debezium and jcustenborder-kafka-config-provider
What I did was I kept the jcustenborder-kafka-config-provider directory structure and copied

All the jar files from debezium into jcustenborder-kafka-config-provider-aws-0.1.2/lib folder
All the .md/.txt files from debezium to jcustenborder-kafka-config-provider-aws-0.1.2/doc folder
and one json files from debezium to jcustenborder-kafka-config-provider-aws-0.1.2 folder

and renamed the folder for e.g. debezium-connector-oracle-v201-SM (SM for secret manager integration) and zipped that to debezium-connector-oracle-v201-SM.zip and uploaded to s3 and created the pluggin. That should work fine.
I was also using GSR (Glue schema registry) for AVRO messages, so I also copied those libs to lib folder.

Jacob Ray Gilman · Answer 11 · Wed Feb 08 2023 07:26:51 GMT+0800 (China Standard Time)

@pc-akothapeta Thank you for the response. I found out the issue is actually because I wasn't creating a NEW custom plugin each time. Based on the AWS console, I was operating under the impression that it would read from the most recent file pointed at in S3, but it seems it does take version into account. By creating a new plugin it immediately resolved my issue, even with my old directory structure...