hafidzdaud / hadoop-overview

Hands on Hadoop, services, installation

Introduction

These are complete notes on Hands on practice, exercises and installation of hadoop related services based on the Hands on Hadoop Course including: HDFS, YARN, MapReduce, Pig, Hive, Ambari, Spark, Mesos, TEZ, HBase, Storm, Oozie, Flink, Scoop, Flume, Kafka, MySQL, Cassandra, MongoDB, Drill, Hue, Phoenix, Presto and Zeppelin.

The live notebook can be found at: axsauze.github.io/hadoop-overview

List of contents:

• Section 1
  • 1-1 - Hortonworks Installation
  • 1-2 - The Hadoop Ecosystem
  • 1-3 - Hadoop Ecosystem and Technologies
• Section 2
  • 2-1 - HDFS: What is it and how it works
  • 2-2 - Install Movielens dataset
  • 2-3 - HDFS: Command line interface
  • 2-4 - MapReduce Fundamental Concepts
  • 2-5 - MapReduce on a Cluster - How MR Scales
  • 2-6 - MapReduce: A Real Life Example
  • 2-7 - Running MapReduce with MRJOB
  • 2-8 - Running with MRJob
  • 2-9 - Hadoop Your Challenge
  • 2-10 - Check your results
• Section 3
  • 3-1 - Introducing Ambari
  • 3-2 - Introducing Pig
  • 3-3 - PIG Example
  • 3-4 - Runing Pig Script
  • 3-5 - PigLatin: Diving Deeper
  • 3-6 - PIG Challenge
• Section 4
  • 4-1 - SPARK
  • 4-2 - Introducing RDDs
  • 4-3 - Example RDD in Spark
  • 4-4 - Spark2.0 way of DataFrames and DataSets
  • 4-5 - Spark2 Finding movies!
  • 4-6 - Using MLLib in Spark
  • 4-7 - Spark2 Challenge
• Section 5
  • 5-1 - What is HIVE?
  • 5-2 - Hive Example
  • 5-3 - How Hive Works
  • 5-4 - Hive Challenge
  • 5-5 - Integrating MySQL & Hadoop
  • 5-6 - Install MySQL and Run Stuff
  • 5-7 - Using Sqoop to extract data from MySQL
  • 5-8 - Sqoop export data to MySQL
• Section 6
  • 6-1 - Why NoSQL
  • 6-2 - HBase
  • 6-3 - Let's play with HBase
  • 6-4 - HBase / Pig Integration
  • 6-5 - Cassandra Overview
  • 6-6 - Installing Cassandra
  • 6-7 - Write Spark to Cassandra
  • 6-8 - MongoDB
  • 6-9 - Installing MongoDB
  • 6-10 - MongoDB Shell
  • 6-11 - Choosing your database
  • 6-12 - Exercise
• Section 7
  • 7-1 - External query engiens
  • 7-2 - Setting up drill
  • 7-3 - Querying with Drill
  • 7-4 - Apache Phoenix
  • 7-5 - Installing Phoenix
  • 7-6 - Using Phoenix
  • 7-7 - What is presto?
  • 7-8 - Installing PRESTO!
  • 7-9 - Presto Cassandra integration
• Section 8
  • 8-1 - Hadoop under the hood
  • 8-2 - TEZ Explained
  • 8-3 - Configure and run hive
  • 8-4 - Apache Mesos
  • 8-5 - Zookeeper Overview
  • 8-6 - Simulating Zookeeper Master Failure
  • 8-7 - Oozie
  • 8-8 - Workflow in Oozie
  • 8-9 - Zeppelin Overview
  • 8-10 - Playing with Zepplin
  • 8-11 - Zeppelin Advanced
  • 8-12 - HUE
  • 8-13 - Other Admin Technologies
• Section 9
  • 9-1 - Streaming with Kafka
  • 9-2 - Setting up Kafka
  • 9-3 - Advanced Kafka
  • 9-4 - Apache Flume
  • 9-5 - Setup Flume
  • 9-6 - Setup Flume Flow
• Section 10
  • 10-1 - Spark STREAMING
  • 10-2 - Spark Streaming Example
  • 10-3 - Python Challenge
  • 10-4 - Apache Storm
  • 10-5 - Apache Storm Example
  • 10-6 - What is Flink
  • 10-7 - Counting words with Flink
• Section 11
  • 11-1 - All the other services
  • 11-2 - How pieces fit together
  • 11-3 - Understanding your requirements
  • 11-4 - Example Top Sellers
  • 11-5 - Example: Movie Recommendations
  • 11-6 - Challenge: System report
• Section 12
  • 12-1 - Books and Online Courses