wgmueller1 / spark-hdfs-tika

spark hdfs tika

build

sbt clean package

run

$ spark-submit --master {URI} \ 
  --jars {path/to/tika-app-1.8.jar} \ 
  target/scala-2.x/spark-hdfs-tika*.jar \ 
  {HDFS_OUTPUT_DIRECTORY} \ 
  {INPUT_FILE} \
  {num of partitions for INPUT_FILE}

getting input list of files from hdfs

$ hadoop fs -ls -R /tmp/data | sed '/^d/ d' | awk '{print substr($0, index($0, $8))}' > input_file.txt