bookeezhou / hadoop-docker

hadoop 3.3.1 docker 安装。干净环境便于测试

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Run Hadoop Cluster within Docker Containers

according above Blog ,update hadoop 3.3.1 version , jdk-8u291-linux-x64.tar.gz version. thanks kiwenlan.

Hadoop Cluster

1. build docker image
download hadoop-3.3.1.tar.gz jdk-8u291-linux-x64.tar.gz file ,
put them into install_tar directory.

chmod +x build-image.sh
./build-image.sh
2. create hadoop network
sudo docker network create --driver=bridge hadoop
3. start container
sudo ./start-container.sh

output:

start hadoop-master container...
start hadoop-slave1 container...
start hadoop-slave2 container...
root@hadoop-master:~#
  • start 3 containers with 1 master and 2 slaves
  • you will get into the /root directory of hadoop-master container
5. start hadoop
./start-hadoop.sh
6. run wordcount
./run-wordcount.sh

output

input file1.txt:
Hello Hadoop

input file2.txt:
Hello Docker

wordcount output:
Docker    1
Hadoop    1
Hello    2

Arbitrary size Hadoop cluster

1. pull docker images and clone github repository

do 1~3 like section A

2. rebuild docker image
sudo ./resize-cluster.sh 5
  • specify parameter > 1: 2, 3..
  • this script just rebuild hadoop image with different slaves file, which pecifies the name of all slave nodes
3. start container
sudo ./start-container.sh 5
  • use the same parameter as the step 2
4. run hadoop cluster

do 5~6 like section A

About

hadoop 3.3.1 docker 安装。干净环境便于测试

License:Apache License 2.0


Languages

Language:Shell 86.7%Language:Dockerfile 13.3%