Giters

Rohit-Jain-2801 / HadoopInstallGuide

Apache Hadoop Components Installation Guide on Windows

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

apachehadoophadoop-ecosysteminstallationinstallation-guidewindowsjavahdfshbasehivepigapache-hbaseapache-hiveapache-pig

HadoopInstallGuide

Apache Hadoop Components Installation Guide on Windows


Apache Hadoop Installation

  • Download Java JDK 8 (v1.8.0_291).
  • Download Hadoop Binary (v3.3.0) Latest Version.
  • Create new folder named Hadoop in the directory where we want to keep all things related to Hadoop & extract hadoop binary in it.
  • Setting Environment Path Variables:
    • Set Variable as JAVA_HOME & Value as <Java Root Path>.
    • Set Variable as HADOOP_HOME & Value as <Hadoop Root Path>.
    • Add following paths to Path Variable:
      • <Java Bin Path>
      • <Hadoop Bin Path>
      • <Hadoop Sbin Path>
  • Check if Java is installed properly by running following commands:
    • javac
    • java -version
  • Make new folder named data in root directory of Hadoop followed by:
    • Making new folder named datanode inside data folder.
    • Making new folder named namenode inside data folder.
  • Make changes in 4 hadoop files located in etc/hadoop/:
    • core-site.xml
    <configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
        <!-- <value>hdfs://0.0.0.0:19000</value> -->
    </property>

    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///E:/Rohit/Hadoop/hadoop/tmp/hadoop-${user.name}<value>
    </property>
</configuration>
    • mapred-site.xml
    <configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>
    • yarn-site.xml
    <configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>  
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
</configuration>
    • hdfs-site.xml
    <configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>

    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///E:/Rohit/Hadoop/hadoop/data/namenode</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///E:/Rohit/Hadoop/hadoop/data/datanode</value>
    </property>
</configuration>
  • Download files for support of Windows & add it to bin folder.
  • Start Hadoop by opening Terminal as Administrator & by running following command:
    • start-all.cmd (or start-dfs.cmd & start-yarn.cmd)
  • Command to check all the Hadoop daemons like DataNode, NameNode, NodeManager & ResourceManager:
    • jps (Java Virtual Machine Process Status Tool)
  • To access Web-UI, open browser & go to:
    • localhost:9870: NameNode Information
    • localhost:9864: DataNode Information
    • localhost:8088: Resource Manager (YARN)
  • Stop Hadoop by running following command:
    • stop-all.cmd (or stop-dfs.cmd & stop-yarn.cmd)

Apache HBase Installation

  • Download HBase Binary (v2.3.5) Latest Version.
  • Preferably extract HBase in the same directory where Hadoop is residing.
  • Make new folders named hbase & zookeeper in root directory of HBase.
  • Open hbase.cmd file placed in <hbase bin> folder &
    • Search for java_arguments as variable.
    • Remove %HEAP_SETTINGS% from the RHS.
  • Open hbase-env.cmd file placed in <hbase conf> folder & add following lines:
set JAVA_HOME=%JAVA_HOME%
set HBASE_CLASSPATH=%HBASE_HOME%\lib\client-facing-thirdparty\*
set HBASE_HEAPSIZE=8000
set HBASE_OPTS="-XX:+UseConcMarkSweepGC" "-Djava.net.preferIPv4Stack=true"
set SERVER_GC_OPTS="-verbose:gc" "-XX:+PrintGCDetails" "-XX:+PrintGCDateStamps" %HBASE_GC_OPTS%
set HBASE_USE_GC_LOGFILE=true

set HBASE_JMX_BASE="-Dcom.sun.management.jmxremote.ssl=false" "-Dcom.sun.management.jmxremote.authenticate=false"

set HBASE_MASTER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10101"
set HBASE_REGIONSERVER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10102"
set HBASE_THRIFT_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10103"
set HBASE_ZOOKEEPER_OPTS=%HBASE_JMX_BASE% "-Dcom.sun.management.jmxremote.port=10104"
set HBASE_REGIONSERVERS=%HBASE_HOME%\conf\regionservers
set HBASE_LOG_DIR=%HBASE_HOME%\logs
set HBASE_IDENT_STRING=%USERNAME%
set HBASE_MANAGES_ZK=true
  • Open hbase-site.xml file placed in <hbase conf> folder & add following lines inside <configuration> tag:
<property>
    <name>hbase.rootdir</name>
    <value>file:///E:/Rohit/Hadoop/HBase/hbase</value>
</property>
<property>
    <name>hbase.zookeeper.property.dataDir</name>
    <value>/E:/Rohit/Hadoop/HBase/zookeeper</value>
</property>
<property>
    <name>hbase.zookeeper.quorum</name>
    <value>localhost</value>
</property>
  • Setting Environment Path Variables:
    • Set Variable as HBASE_HOME & Value as <HBase Root Path>.
    • Set Variable as HBASE_BIN_PATH & Value as <HBase Bin Path>.
    • Add <HBase Bin Path> path to Path Variable.
  • Start HBase by opening Terminal as Administrator & by running following commands:
    • start-all.cmd (Hadoop)
    • start-hbase.cmd (HBase)
  • To interact with HBase, run following command: hbase shell.
  • Start HBase by running following command: stop-hbase.cmd.

Apache Hive Installation

  • Download Relational Database - Apache Derby Binary (v10.14.2.0) Latest Version to create its Metastore (where all metadata will be stored).
  • Preferably extract Derby in the same directory where Hadoop is residing.
  • Download Cygwin (v3.2.0) Latest Version & Install it.
  • Download Hive Binary (v3.1.2) Latest Version.
  • Preferably extract Hive in the same directory where Hadoop is residing.
  • Setting Environment Path Variables:
    • Set Variable as HIVE_HOME & Value as <Hive Root Path>.
    • Set Variable as DERBY_HOME & Value as <Dirby Root Path>.
    • Set Variable as HIVE_LIB & Value as <Hive Lib Path>.
    • Set Variable as HIVE_BIN & Value as <Hive Bin Path>.
    • Set Variable as HADOOP_USER_CLASSPATH_FIRST & Value as true.
    • Add following paths to Path Variable:
      • <Dirby Bin Path>
      • <Hive Bin Path>
  • Copy files from Derby Lib folder to Hive Lib folder.
  • Create a new file named hive-site.xml in <hive conf> folder & add following lines:
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
	<property>
		<name>javax.jdo.option.ConnectionURL</name> 
		<value>jdbc:derby://localhost:1527/metastore_db;create=true</value> 
		<description>JDBC connect string for a JDBC metastore</description>
	</property>
	<property> 
		<name>javax.jdo.option.ConnectionDriverName</name> 
		<value>org.apache.derby.jdbc.ClientDriver</value> 
		<description>Driver class name for a JDBC metastore</description>
	</property>
	<property> 
		<name>hive.server2.enable.doAs</name> 
		<description>Enable user impersonation for HiveServer2</description>
		<value>true</value>
	</property>
	<property>
		<name>hive.server2.authentication</name> 
		<value>NONE</value>
		<description> Client authentication types. NONE: no authentication check LDAP: LDAP/AD based authentication KERBEROS: Kerberos/GSSAPI authentication CUSTOM: Custom authentication provider (Use with property hive.server2.custom.authentication.class) </description>
	</property>
	<property>
		<name>datanucleus.autoCreateTables</name>
		<value>True</value>
	</property>
</configuration>
  • Download extra cmd files for Windows support from this & replace in Hive bin directory along with sub-directories.
  • Replace the Hive guava-19.0.jar stored in Hive Lib with Hadoop’s guava-27.0-jre.jar found in hadoop\share\hadoop\hdfs\lib.
  • Make new directories in following locations as:
    • E:\cygdrive
    • C:\cygdrive
  • Open the Terminal as Administrator and execute the following commands for symbolic links:
    • mklink /J E:\cygdrive\e\ E:\
    • mklink /J C:\cygdrive\c\ C:\
  • Start Derby by opening Terminal as Administrator & by running following command: StartNetworkServer -h 0.0.0.0
  • Open Cygwin utility and execute the following command: cygstart ~/.bashrc & add following lines:
export HADOOP_HOME='/cygdrive/e/Rohit/Hadoop/hadoop'
export PATH=$PATH:$HADOOP_HOME/bin
export HIVE_HOME='/cygdrive/e/Rohit/Hadoop/hive'
export PATH=$PATH:$HIVE_HOME/bin
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HIVE_HOME/lib/*.jar
  • Comment 2 lines in file hive-schema-3.1.0.derby.sql in hive\scripts\metastore\upgrade\derby folder containing:
    • Line 1: CREATE FUNCTION "APP"."NUCLEUS_ASCII" ...
    • Line 2: CREATE FUNCTION "APP"."NUCLEUS_MATCHES" ...
  • Inside Cygwin utility, goto hive-bin by cd $HIVE_HOME/bin & run command: schematool -dbType derby -initSchema for Initializing Hive Metastore.
  • Start Hive by opening Terminal as Administrator & by running following commands:
    • start-all.cmd (Hadoop)
    • hadoop dfsadmin -safemode leave (Disabling SafeMode of Hadoop)
    • hive --service hiveserver2 start (HiveServer2 service)
    • hive (Apache Hive)

Apache Pig Installation

  • Download Pig Binary (v0.17.0) Latest Version.
  • Note: The Apache Pig v0.17.0 supports Hadoop 2.x versions & it is facing some compatibility issues with Hadoop 3.x.
  • Preferably extract Pig in the same directory where Hadoop is residing.
  • Setting Environment Path Variables:
    • Set Variable as PIG_HOME & Value as <Pig Root Path>.
    • Add following path to Path Variable: <Pig Bin Path>
  • Make change of HADOOP_BIN_PATH from %HADOOP_HOME%\bin to %HADOOP_HOME%\libexec in pig.cmd file located in Pig Bin folder.
  • To check if pig is installed properly, run command: pig -version.
  • The PigLatin statements can be run in two ways:
    • Local: All scripts are executed on a single machine without requiring Hadoop. (command: pig -x local)
    • MapReduce: Scripts are executed on a Hadoop cluster (command: pig -x MapReduce)

Note:

  • All the customs paths mentioned above has to be configured according your own system.
  • All the above installation steps are collected from various source available on internet. I have just cumulated them together here.
  • This guide may not be updated for later versions or other components of Apache.
  • If there is any issues, please contact through Email or If you want to contribute, create a pull request.

About

Apache Hadoop Components Installation Guide on Windows

apachehadoophadoop-ecosysteminstallationinstallation-guidewindowsjavahdfshbasehivepigapache-hbaseapache-hiveapache-pig