Kerberized HDFS Support

Question

Kerberized HDFS Support

yomiel-s opened this issue 4 years ago · comments

Hi,juicesync is an awsome data sync tool.But I found that when I start to sync data from 2 hdfss,it shows "no available namenodes: write tcp x.x.x.x:59738->x.x.x.x:8020: write: broken pipe".Does it support kerberized HDFS?Thank you.

Davies Liu · Answer 1 · Mon Dec 14 2020 15:45:56 GMT+0800 (China Standard Time)

@HooinKyoma Well received, the library [1] used here does support kerberized HDFS, will support it soon.

[1] github.com/colinmarc/hdfs

Davies Liu · Answer 2 · Mon Dec 14 2020 16:38:05 GMT+0800 (China Standard Time)

@HooinKyoma Kerberized HDFS is supported now, please build from master branch, or wait for a new release.

Davies Liu · Answer 3 · Mon Dec 14 2020 16:41:38 GMT+0800 (China Standard Time)

@HooinKyoma wait, it has difficulty to talk to data node, we are working on it.

Davies Liu · Answer 4 · Mon Dec 14 2020 16:44:46 GMT+0800 (China Standard Time)

related to this one: colinmarc/hdfs#248

Yomiel · Answer 5 · Mon Dec 14 2020 16:53:38 GMT+0800 (China Standard Time)

Got it.Thank you for your hard work!

Davies Liu · Answer 6 · Mon Dec 14 2020 17:52:50 GMT+0800 (China Standard Time)

Created colinmarc/hdfs#258

Davies Liu · Answer 7 · Thu Dec 31 2020 11:16:26 GMT+0800 (China Standard Time)

juicesync had upgrade HDFS client to 2.2.0 to address this issue, so Kerberized HDFS is fully supported. cc @HooinKyoma

Davies Liu · Answer 8 · Thu Dec 31 2020 11:16:57 GMT+0800 (China Standard Time)

@HooinKyoma Please try juicesync v0.6.1