data-platform-hq / terraform-azurerm-adls-v2

Terraform module for managing Azure Datalake Storage gen2

Azure Data Lake Storage Gen2 Terraform module

Terraform module for creation Azure Data Lake Storage Gen2 file system

Usage

By default, this module will provision Data Lake Gen2 Path in a File System within an Azure Storage Account with custom Bash script, which does not require any Storage specific role. (E.x., Storage Blob Data Contributor/Owner).

If your Service Principal for Terraform is assigned to "Storage Blob Data Owner" role it is possible to create File System folders with Azurerm resource(azurerm_storage_data_lake_gen2_path). In this case, please set "storage_role_assigned" variable to 'true'.

Below is an example of module usage, here you are providing users in "adls_ro_users" and "adls_rw_users" variables. After, passing this map to ADLS module.

locals {
  # Map with adls2 file systems configuration
  adls2fs = {
    "my-data-lake" = {
      permissions = [
        { group = "********-****-****-****-************", permissions = "rwx", scope = "access", type = "group" },
        { group = "********-****-****-****-************", permissions = "---", scope = "default", type = "group" },
        { group = "********-****-****-****-************", permissions = "rwx", scope = "access", type = "group" },
        { group = "********-****-****-****-************", permissions = "---", scope = "default", type = "group" }
      ]
      folders_config = [
        {
          path: "raw",
          permissions: [
            { group = "********-****-****-****-************", permissions = "r-x", scope = "access", type = "group" },
            { user = "********-****-****-****-************", permissions = "rwx", scope = "access", type = "user" },
            { group = "********-****-****-****-************", permissions = "r-x", scope = "default", type = "group" },
          ]
        },
        {
          path: "e2e",
          permissions: [
            { group = "********-****-****-****-************", permissions = "r-x", scope = "access", type = "group" },
            { user = "********-****-****-****-************", permissions = "rwx", scope = "access", type = "user" },
            { group = "********-****-****-****-************", permissions = "r-x", scope = "default", type = "group" }
          ]
        },
        {
          path: "test",
          permissions: []
        }
      ]
    }
  }
}

module "adls" {
  source   = "data-platform-hq/adls-v2/azurerm"
  for_each = local.adls2fs
  
  name                  = each.key
  permissions           = each.value.permissions
  storage_role_assigned = true 
  folders_config        = each.value.folders_config
  storage_account_id    = module.storage_account.id
  storage_account_name  = module.storage_account.name
  properties            = {
    env = "Development"
  }
}

Requirements

Name	Version
terraform	>= 1.0.0
azurerm	>= 3.23.0
null	>=3.1.1

Providers

Name	Version
azurerm	>= 3.23.0
null	>=3.1.1

Modules

No modules.

Resources

Name	Type
azurerm_storage_data_lake_gen2_filesystem.this	resource
azurerm_storage_data_lake_gen2_path.other	resource
null_resource.create_folders	resource

Inputs

Name	Description	Type	Default	Required
ace_default	Default ACE values	list(map(string))	[ { "permissions": "---", "scope": "access", "type": "other" }, { "permissions": "---", "scope": "default", "type": "other" }, { "permissions": "rwx", "scope": "access", "type": "group" }, { "permissions": "rwx", "scope": "access", "type": "mask" }, { "permissions": "rwx", "scope": "access", "type": "user" }, { "permissions": "rwx", "scope": "default", "type": "group" }, { "permissions": "rwx", "scope": "default", "type": "mask" }, { "permissions": "rwx", "scope": "default", "type": "user" } ]	no
folders_config	List of ADLS folders configuration to create	list(object({ path = string permissions = any }))	[]	no
name	Name of ADLS FS to create	string	n/a	yes
permissions	List of ADLS FS permissions	list(map(string))	[ {} ]	no
properties	Map of properties	map(string)	{}	no
storage_account_id	ID of storage account to create ADLS in	string	n/a	yes
storage_account_name	Name of storage account to create ADLS in	string	n/a	yes
storage_role_assigned	Is Storage Blob Data Owner Role assigned to Terraform Service Principal? Provides an ability to create File System with bash script(false) or azurerm resources(true).	bool	false	no

Outputs

Name	Description
id	The ID of the Data Lake Storage Gen2 Filesystem (container ID).
name	The name of the Data Lake Storage Gen2 Filesystem (container name).
storage_account_id	The ID of the Storage Account where the Data Lake Storage Gen2 Filesystem exists.

License

Apache 2 Licensed. For more information please see LICENSE