kumarvna / terraform-azurerm-hdinsight

Terraform module to create managed, full-spectrum, open-source analytics service Azure HDInsight. This module creates Apache Hadoop, Apache Spark, Apache HBase, Interactive Query (Apache Hive LLAP) and Apache Kafka clusters.

Azure HDInsight Terraform Module

Terraform module to create managed, full-spectrum, open-source analytics service Azure HDInsight. This module creates Apache Hadoop, Apache Spark, Apache HBase, Interactive Query (Apache Hive LLAP) and Apache Kafka clusters.

Resources supported

• Apache Spark Cluster
• Apache Hadoop Cluster
• Apache HBase Cluster
• Apache Hive LLAP (Interactive Query) Cluster
• Apache Kafka Cluster

Module Usage for HDInsight Hadoop Cluster

# Azurerm Provider configuration
provider "azurerm" {
  features {
  }
}

module "hdinsight" {
  source  = "kumarvna/hdinsight/azurerm"
    version = "1.0.0"

  # By default, this module will not create a resource group. Location will be same as existing RG.
  # proivde a name to use an existing resource group, specify the existing resource group name, 
  # set the argument to `create_resource_group = true` to create new resrouce group.
  resource_group_name = "rg-shared-westeurope-01"
  location            = "westeurope"

  # The type of hdinsight cluster to create 
  # Valid values are `hadoop`, `hbase`, `interactive_query`, `kafka`, `spark`.
  hdinsight_cluster_type = "hadoop"

  # Hdinsight hadoop cluster configuration. Gateway credentials must be different from the one used 
  # for the `head_node`, `worker_node` and `zookeeper_node` roles.
  hadoop_cluster = {
    name             = "hadoopdemocluster1"
    cluster_version  = "3.6"
    gateway_username = "acctestusrgw"
    gateway_password = "TerrAform123!"
    hadoop_version   = "2.7"
    tier             = "Standard"
  }

  # Node configuration
  # Either a password or one or more ssh_keys must be specified - but not both.
  # Password must be at least 10 characters in length and must contain digits,uppercase, 
  # lower case letters and non-alphanumeric characters 
  hadoop_roles = {
    vm_username = "acctestusrvm"
    vm_password = "AccTestvdSC4daf986!"
    head_node = {
      vm_size = "Standard_D3_V2"
    }
    worker_node = {
      vm_size               = "Standard_D4_V2"
      target_instance_count = 3
      autoscale = {
        capacity = {
          max_instance_count = 4
          min_instance_count = 3
        }
      }
    }
    zookeeper_node = {
      vm_size = "Standard_D3_V2"
    }
  }

  # Use Azure Monitor logs to monitor HDInsight clusters. Recommended to place both the HDInsight 
  # cluster and the Log Analytics workspace in the same region for better performance.
  enable_hadoop_monitoring     = true
  log_analytics_workspace_name = "loganalytics-we-sharedtest2"

  # Tags for Azure Resources
  tags = {
    Terraform   = "true"
    Environment = "dev"
    Owner       = "test-user"
  }
}

Module Usage examples for

• Hdinsight Hadoop Cluster
• Hdinsight HBase Cluster
• Hdinsight Interactive Query Cluster
• Hdinsight Kafka Cluster
• Hdinsight Spark Cluster

Requirements

Name	Version
terraform	>= 0.13
azurerm	>= 2.59.0

Providers

Name	Version
azurerm	>= 2.59.0
random	>= 3.1.0

Inputs

Name	Description	Type	Default
create_resource_group	Whether to create resource group and use it for all networking resources	string	"false"
resource_group_name	The name of the resource group in which resources are created	string	""
location	The location of the resource group in which resources are created	string	""
storage_account_name	The name of the storage account	string	null
storage_container_name	The name of the storage container	string	``null`
storage_account	Manages an Azure Storage Account configuration	object	{}
hdinsight_cluster_type	Specify the type of hdinsight clusters. Valid values arehadoop,hbase,interactive-query,kafka,spark	string	null
hadoop_cluster	Manages a HDInsight Hadoop Cluster	object	null
hadoop_roles	Manage node roles for HDInsight Hadoop Cluster	object	null
hadoop_storage_account_gen2	Specify configuration for Azure Data Lake Storage Gen 2	object	null
hadoop_network	Specify the networking and private endpoint configuration	object	null
hadoop_metastores	HDInsight metadata with external data stores, available forApache Hive metastore,Apache Oozie metastore, andApache Ambari database	object	null
enable_hadoop_monitoring	Use Azure Monitor logs to monitor HDInsight hadoop cluster	string	false
log_analytics_workspace_name	The name of log analytics workspace name	string	null
hbase_cluster	Manages a HDInsight HBase Cluster	object	null
hbase_roles	Manage node roles for HDInsight HBase Cluster	object	null
hbase_storage_account_gen2	Specify configuration for Azure Data Lake Storage Gen 2	object	null
hbase_metastores	HDInsight metadata with external data stores, available forApache Hive metastore,Apache Oozie metastore, andApache Ambari database	object	null
enable_hbase_monitoring	Use Azure Monitor logs to monitor HDInsight HBase cluster	string	false
kafka_cluster	Manages a HDInsight Kafka Cluster	object	null
kafka_roles	Manage node roles for HDInsight Kafka Cluster	object	null
kafka_storage_account_gen2	Specify configuration for Azure Data Lake Storage Gen 2	object	null
kafka_metastores	HDInsight metadata with external data stores, available forApache Hive metastore,Apache Oozie metastore, andApache Ambari database	object	null
enable_kafka_monitoring	Use Azure Monitor logs to monitor HDInsight Kafka cluster	string	false
spark_cluster	Manages a HDInsight Spark Cluster	object	null
spark_roles	Manage node roles for HDInsight Spark Cluster	object	null
spark_storage_account_gen2	Specify configuration for Azure Data Lake Storage Gen 2	object	null
spark_network	Specify the networking and private endpoint configuration	object	null
spark_metastores	HDInsight metadata with external data stores, available forApache Hive metastore,Apache Oozie metastore, andApache Ambari database	object	null
enable_spark_monitoring	Use Azure Monitor logs to monitor HDInsight Spark cluster	string	false
interactive_query_cluster	Manages a HDInsight Interactive Query Cluster	object	null
interactive_query_roles	Manage node roles for HDInsight Interactive Query Cluster	object	null
interactive_query_storage_account_gen2	Specify configuration for Azure Data Lake Storage Gen 2	object	null
interactive_query_network	Specify the networking and private endpoint configuration	object	null
interactive_query_metastores	HDInsight metadata with external data stores, available forApache Hive metastore,Apache Oozie metastore, andApache Ambari database	object	null
enable_interactive_query_monitoring	Use Azure Monitor logs to monitor HDInsight Interactive Query cluster	string	false
tags	A map of tags to add to all resources	map	{}

Outputs

Name	Description
hadoop_cluster_id	The ID of the HDInsight Hadoop Cluster
hadoop_cluster_https_endpoint	The HTTPS Connectivity Endpoint for this HDInsight Hadoop Cluster
hadoop_cluster_ssh_endpoint	The SSH Connectivity Endpoint for this HDInsight Hadoop Cluster
hbase_cluster_id	The ID of the HDInsight HBase Cluster
hbase_cluster_https_endpoint	The HTTPS Connectivity Endpoint for this HDInsight HBase Cluster
hbase_cluster_ssh_endpoint	The SSH Connectivity Endpoint for this HDInsight HBase Cluster
kafka_cluster_id	The ID of the HDInsight Kafka Cluster
kafka_cluster_https_endpoint	The HTTPS Connectivity Endpoint for this HDInsight Kafka Cluster
kafka_cluster_ssh_endpoint	The SSH Connectivity Endpoint for this HDInsight Kafka Cluster
spark_cluster_id	The ID of the HDInsight Spark Cluster
spark_cluster_https_endpoint	The HTTPS Connectivity Endpoint for this HDInsight Spark Cluster
spark_cluster_ssh_endpoint	The SSH Connectivity Endpoint for this HDInsight Spark Cluster
interactive_query_cluster_id	The ID of the HDInsight Interactive Query Cluster
interactive_query_cluster_https_endpoint	The HTTPS Connectivity Endpoint for this HDInsight Interactive Query Cluster
interactive_query_cluster_ssh_endpoint	The SSH Connectivity Endpoint for this HDInsight Interactive Query Cluster

Resource Graph

Authors

Originally created by Kumaraswamy Vithanala

Other resources

• Azure HDInsight
• Terraform AzureRM Provider Documentation