demon386 / pdoop

A golang written utilities to get files from a HDFS directory in parallel

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

pdoop

A golang written utilities to get files from a HDFS directory in parallel. In my working environment it's about 3.8 times faster than using hadoop fs -getmerge when merging as a single file.

It relies on $HADOOP_HOME environmental variable.

Usage:

pdoop hdfs_dir local_path

Options:

  • -m: merge hdfs dir as a single file. It's disabled by default. NOTE: when merging as a single file, the order of different files are not guaranteed.
  • -p: number of parallel downloads, 10 by default.

About

A golang written utilities to get files from a HDFS directory in parallel


Languages

Language:Go 100.0%