qf6101 / topwords

Implementation of paper: Deng K, Bol P K, Li K J, et al. On the unsupervised analysis of domain-specific Chinese texts[J]. Proceedings of the National Academy of Sciences, 2016: 201516510.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

/opt/cloudera/parcels/CDH/lib/hadoop/bin/yarn: No such file or directory

wanghaisheng opened this issue · comments

i am using cdh integrated spark
and config like this

#!/usr/bin/env bash

# get into the current directory
cd "$( cd "$( dirname "$0"  )" && pwd  )"

##### The Parameters You Need to Predefine Start #####

# set the environment variables
#HADOOP_HOME="/path/to/hadoop"  #Hadoop home
HADOOP_HOME="/opt/cloudera/parcels/CDH/lib/hadoop"
SPARK_HOME="/opt/cloudera/parcels/CDH/lib/spark"
#SPARK_HOME="/path/to/spark"  #Spark home:
topwords_jar="../release/topwords-1.0.jar"  #topwords jar file
...............with no changes

[docker@node3 topwords]$ bash deploy/sbin/topwords_yarn.sh
deploy/sbin/topwords_yarn.sh: line 65: /opt/cloudera/parcels/CDH/lib/hadoop/bin/yarn: No such file or directory


[docker@node3 topwords]$ cat /etc/hadoop/conf/mapred-site.xml | grep yarn
    <value>yarn</value>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <name>yarn.app.mapreduce.am.admin.user.env</name>
[docker@node3 topwords]$ cat  /etc/hadoop/conf.cloudera.yarn/yarn-site.xml  | grep 80
    <value>node3:8032</value>
    <value>node3:8033</value>
    <value>node3:8030</value>
    <value>node3:8031</value>
    <value>node3:8088</value>
    <value>node3:8090</value>
[docker@node3 topwords]$ cat  /etc/hadoop/conf.cloudera.yarn/yarn-site.xml  | grep DIR
    <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>

I used Apache Hadoop instead of CDH. I guess you're supposed to check the corresponding command wrt '${HADOOP_HOME}/bin/yarn logs -applicationId' in CDH.