HDP installation automation (AWS+SaltStack+Blueprints)

Introduction

This document describes how to automate HDP installation on AWS.

Servers preparation

AWS ec2 instances

Create following type of servers (ec2 instances with Amazon Linux AMI):

Salt Master - server, which serve as main orchestration server. Use minimal configuration - t2.micro). It is mandatory component.
Ambari Server - server, which hosts Ambari components. t2.medium
Master + Data Node - Set of servers, which serve as master and data node. For example, 4r.large. It depends on performed tasks and installed components

Ambari server and master/data nodes can be installed all on one machine. But such configuration can be used only for learning purposes.

Give the name of servers (as tags) in AWS console: salt-master, ambari-server, master-01, data-01, data-02, etc.

TODO: Add picture

Set hostnames

Set appropriate hostnames for all servers. It should be hostname+domain. For example: salt-master.test

In addition add line "IP-ADDRESS-OF-SALTMASTER salt" to /etc/hosts.

This is manual operation, which must be executed on all servers. Use this code:

sudo sed -i "s/HOSTNAME=.*/HOSTNAME=[hostname.domain]/g" /etc/sysconfig/network
echo '[saltserver-ip] salt' | sudo tee --append /etc/hosts
sudo reboot

Install salt-minions

After all servers are rebooted, run on all nodes, except salt-master

curl -L https://bootstrap.saltstack.com -o install_salt.sh
sudo sh install_salt.sh

After minion is installed, we don't need to access manually machines with salt-minions by ssh.

Install salt-master

Install SaltStack master on salt-master machine.

curl -L https://bootstrap.saltstack.com -o install_salt.sh
sudo sh install_salt.sh -M

Install git and get all necessary artefacts (salt files and Ambari blueprints) from repository:

sudo yum -y install git
sudo git clone https://github.com/vdovychenko/hadoop-automation-blueprints.git /srv
sudo chmod 646 /srv/pillar/ambari.sls

Install AWS python library (boto3) for automating some part:

sudo pip install boto3

Accept all salt-minions to be manageable from salt-master:

sudo salt-key -A

and test if they are pingable:

sudo salt '*' test.ping

Set AWS nodes private IP in pillar file, so salt knows their IP addresses:

export AWS_DEFAULT_REGION=us-east-1
python /srv/aws-api.py

Manual Settings

Detailed setting can be done in following files:

pillar file - hostnames
blueprint - distribution of hadoop components across nodes

Pillar file is located: /srv/pillar/ambari.sls

ambari:
  # The cluster domain name
  domain: cap

  # The short host name where Ambari will be installed
  ambarinode: master-01

  # Blueprint Name, if changed, make sure there are blueprint-[...].json and hosts-[...].json
  blueprintname: hdp

nodes:
  master-01: 172.31.56.130
  data-01:   172.31.63.206
  data-02:   172.31.58.111

Blueprint file is located: /srv/salt/blueprint/hosts-[blueprintname].json

Run HDP installation

On salt-server run following command to start installation:

sudo salt '*' state.apply

It will install:

Ambari server
Ambari agents
run HDP installation

vdovychenko / hadoop-automation-blueprints