confluentinc / kafka-connect-hdfs

Kafka Connect HDFS connector

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

OzoneFileSystem

tothandor opened this issue · comments

I try to use an Ozone filesystem with this HDFS Sink Connector, but I get the following error, which is weird since OzoneFileSystem is a subclass of Hadoop FileSystem.

I could use this very filesystem with CLI commands like hdfs dfs -ls ofs://... or through libhdfs from anywhere else (eg. pyarrow.fs.HadoopFilesystem.from_uri("ofs://..."))), so I would suppose that it should be usable natively in Java.

I am not a Java developer, but I tried my best to interpret the sources of the involved libraries to find out what is the problem, but I can't figure it out. Maybe this is not the problem of this connector, but it seems to me, that the other HdfsStorage constructor should be invoked, to make it work.

Caused by: java.lang.ClassCastException: org.apache.hadoop.fs.ozone.OzoneFileSystem cannot be cast to org.apache.hadoop.fs.FileSystem
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3260)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:121)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3310)
        at org.apache.hadoop.fs.FileSystem$Cache.getUnique(FileSystem.java:3284)
        at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:528)
        at io.confluent.connect.hdfs.storage.HdfsStorage.<init>(HdfsStorage.java:60)

Please help me to find this out!