Giters

confluentinc / kafka-connect-hdfs

Kafka Connect HDFS connector

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Recent releases not in confluent repo

JozoVilcek opened this issue · comments

commented

I can not find recent releases in confluent package repo:
http://packages.confluent.io/maven/io/confluent/kafka-connect-hdfs/

Last available is 5.2.4 but latest release is 10.0.0. Is this expected? Is there a way how to consume recent connector release as maven like dependency?

commented

Hi @JozoVilcek
What is missing is the updated index on that url. That prevents us from seeing the new artifacts on the web browser. But they are still discoverable via maven. So basically, you can add the right version as a maven dependency in your pom and you'll be able to fetch the artifacts. You can see the existing versions in https://www.confluent.io/hub/confluentinc/kafka-connect-hdfs or search for a release tag: https://github.com/confluentinc/kafka-connect-hdfs/tags

This is an internal tooling restriction at the moment. But again, it does not prevent using the publicly available released artifacts as dependencies.

commented

Thanks @kkonstantine for quick reply. Yeah, as I did not see the version I even did not try to run it. It works fine for me. Thanks!

commented

@kkonstantine appologies for chaos. I was too quick to test and was hitting my local cache. The dependency does not get resolved for me.

git clone https://github.com/JozoVilcek/confluent-dependency-test.git
cd confluent-dependency-test
./gradlew build

yields

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not resolve all files for configuration ':compileClasspath'.
   > Could not find io.confluent:kafka-connect-hdfs:10.0.0.
     Searched in the following locations:
       - https://repo.maven.apache.org/maven2/io/confluent/kafka-connect-hdfs/10.0.0/kafka-connect-hdfs-10.0.0.pom
       - http://packages.confluent.io/maven/io/confluent/kafka-connect-hdfs/10.0.0/kafka-connect-hdfs-10.0.0.pom

What am I doing wrong?

commented

v10.0.0 is indeed missing for hdfs and will need to be restored.

However, v10.0.1 has just been released and should be addressable via maven. Want to try this one instead?

commented

Yes, version 10.0.1 works. I can use the newer version. Feel free to close this issue or keep it open for the 10.0.0 if you like. Thanks!

commented

10.0.0 is not the only release missing, most (all?) recent releases are missing and cannot be loaded with maven. Currently the only way to get recent releases is confluent hub or building from source. AFAIK what is seen with browser is exactly what is available with maven as well.

commented

Hi @Jootoi could you specify the versions you're seeing an issue with?

commented

Tested not working: 10.0.2, 10.0.3, 10.0.4, 10.0.5, 10.0.6, 10.0.7, 5.5.4, 5.3.5
Tested working: 10.0.1, 5.2.5
Not testing more because the pattern is pretty clear, if it is not visible in http://packages.confluent.io/maven/io/confluent/kafka-connect-hdfs/ with browser, it cannot be loaded with maven.

Tested using this pom.xml: https://gist.github.com/Jootoi/d824234d4276b372c372b19c755ff352
Failing versions failed with error:
Failed to execute goal on project confluent-dependency-test: Could not resolve dependencies for project org.example:confluent-dependency-test:jar:1.0-SNAPSHOT: Could not find artifact io.confluent:kafka-connect-hdfs:jar:5.3.5 in confluent (http://packages.confluent.io/maven/)

Working version failed when trying to load org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.5-jhyde. I guess that is a separate problem because I have not set up valid repo for pentaho dependencies.