Giters

mineiros-io / terraform-google-composer

A Terraform module to create and manage Cloud Composer, a fully managed workflow orchestration service built on Apache Airflow, in Google Cloud https://cloud.google.com/composer

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

mineirosiacterraform-modulesterraform

Build Status GitHub tag (latest SemVer) Terraform Version Google Provider Version Join Slack

terraform-google-composer

A Terraform module to create a Google Cloud Composer on Google Cloud Services (GCP).

This module supports Terraform version 1 and is compatible with the Terraform Google Provider version 4.

This module is part of our Infrastructure as Code (IaC) framework that enables our users and customers to easily deploy and manage reusable, secure, and production-grade cloud infrastructure.

Module Features

A Terraform base module for creating a terraform-google-composer. Google Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow.

Composer Versions

Google Cloud Composer has two major versions: Cloud Composer 1 and Cloud Composer 2. Some new Cloud Composer features might be supported only in Cloud Composer 2. For details please see: https://cloud.google.com/composer/docs/composer-2/composer-versioning-overview

Getting Started

Most common usage of the module just setting required arguments:

module "terraform-google-composer" {
  source = "git@github.com:mineiros-io/terraform-google-composer.git?ref=v0.2.0"

  name   = "example-name"

  software_config = {
    python_version = 3
  }
}

Module Argument Reference

See variables.tf and examples/ for details and use-cases.

Top-level Arguments

Main Resource Configuration

  • name: (Required string)

    The location or Compute Engine region for the environment.

  • project: (Required string)

    The ID of the project in which the resource belongs. If it is not provided, the provider project is used.

  • region: (Optional string)

    The location or Compute Engine region for the environment.

  • labels: (Optional map(string))

    User-defined labels for this environment. The labels map can contain no more than 64 entries. Entries of the labels map are UTF8 strings that comply with the following restrictions: Label keys must be between 1 and 63 characters long and must conform to the following regular expression: [a-z]([-a-z0-9]*[a-z0-9])?. Label values must be between 0 and 63 characters long and must conform to the regular expression ([a-z]([-a-z0-9]*[a-z0-9])?)?. No more than 64 labels can be associated with a given environment. Both keys and values must be <= 128 bytes in size.

  • node_count: (Optional number)

    The number of nodes in the Kubernetes Engine cluster that will be used to run this environment. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

  • software_config: (Optional object(software_config))

    The configuration settings for software inside the environment.

    The software_config object accepts the following attributes:

    • airflow_config_overrides: (Optional map(string))

      Apache Airflow configuration properties to override. Property keys contain the section and property names, separated by a hyphen, for example "core-dags_are_paused_at_creation".

      Section names must not contain hyphens ("-"), opening square brackets ("["), or closing square brackets ("]"). The property name must not be empty and cannot contain "=" or ";". Section and property names cannot contain characters: "." Apache Airflow configuration property names must be written in snake_case. Property values can contain any character, and can be written in any lower/upper case format. Certain Apache Airflow configuration property values are blacklisted, and cannot be overridden.

    • pypi_packages: (Optional map(string))

      Custom Python Package Index (PyPI) packages to be installed in the environment. Keys refer to the lowercase package name (e.g. "numpy"). Values are the lowercase extras and version specifier (e.g. "==1.12.0", "[devel,gcp_api]", "[devel]>=1.8.2, <1.9.2"). To specify a package without pinning it to a version specifier, use the empty string as the value.

    • env_variables: (Optional map(string))

      Additional environment variables to provide to the Apache Airflow scheduler, worker, and webserver processes. Environment variable names must match the regular expression [a-zA-Z_][a-zA-Z0-9_]*. They cannot specify Apache Airflow software configuration overrides (they cannot match the regular expression AIRFLOW__[A-Z0-9_]+__[A-Z0-9_]+), and they cannot match any of the following reserved names:

      AIRFLOW_HOME,
C_FORCE_ROOT,
CONTAINER_NAME,
DAGS_FOLDER,
GCP_PROJECT,
GCS_BUCKET,
GKE_CLUSTER_NAME,
SQL_DATABASE,
SQL_INSTANCE,
SQL_PASSWORD,
SQL_PROJECT,
SQL_REGION,
SQL_USER

    • image_version: (Optional string)

      The version of the software running in the environment. This encapsulates both the version of Cloud Composer functionality and the version of Apache Airflow. It must match the regular expression composer-[0-9]+\.[0-9]+(\.[0-9]+)?-airflow-[0-9]+\.[0-9]+(\.[0-9]+.*)?. The Cloud Composer portion of the version is a semantic version. The portion of the image version following 'airflow-' is an official Apache Airflow repository release name.

    • python_version: (Optional string)

      The major version of Python used to run the Apache Airflow scheduler, worker, and webserver processes. Can be set to '2' or '3'. If not specified, the default is '3'.

    • scheduler_count: (Optional number)

      Cloud Composer 1 with Airflow 2 only. The number of schedulers for Airflow. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-2.*.*.

  • private_environment_config: (Optional object(private_environment_config))

    The configuration used for the Private IP Cloud Composer environment.

    The private_environment_config object accepts the following attributes:

    • enable_private_endpoint: (Optional bool)

      If true, access to the public endpoint of the GKE cluster is denied. If this field is set to true, ip_allocation_policy.use_ip_aliases must be set to true for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • master_ipv4_cidr_block: (Optional string)

      The IP range in CIDR notation to use for the hosted master network. This range is used for assigning internal IP addresses to the cluster master or set of masters and to the internal load balancer virtual IP. This range must not overlap with any other ranges in use within the cluster's network.

      Default is "172.16.0.0/28".

    • cloud_sql_ipv4_cidr_block: (Optional string)

      The CIDR block from which IP range in tenant project will be reserved for Cloud SQL. Needs to be disjoint from web_server_ipv4_cidr_block

    • web_server_ipv4_cidr_block: (Optional string)

      The CIDR block from which IP range for web server will be reserved. Needs to be disjoint from master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • web_server_ipv4_cidr_block: (Optional string)

      The CIDR block from which IP range for web server will be reserved. Needs to be disjoint from master_ipv4_cidr_block and cloud_sql_ipv4_cidr_block. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

  • database_machine_type: (Optional string)

    Cloud SQL machine type used by Airflow database. It has to be one of: db-n1-standard-2, db-n1-standard-4, db-n1-standard-8 or db-n1-standard-16.

  • webserver_machine_type: (Optional string)

    Machine type on which Airflow web server is running. It has to be one of: composer-n1-webserver-2, composer-n1-webserver-4 or composer-n1-webserver-8. Value custom is returned only in response, if Airflow web server parameters were manually changed to a non-standard values.

  • web_server_allowed_ip_ranges: (Optional list(web_server_allowed_ip_range))

    A collection of allowed IP ranges with descriptions.

    Each web_server_allowed_ip_range object in the list accepts the following attributes:

    • value: (Required string)

      IP address or range, defined using CIDR notation, of requests that this rule applies to. Examples: 192.168.1.1 or 192.168.0.0/16 or 2001:db8::/32 or 2001:0db8:0000:0042:0000:8a2e:0370:7334. IP range prefixes should be properly truncated. For example, 1.2.3.4/24 should be truncated to 1.2.3.0/24. Similarly, for IPv6, 2001:db8::1/32 should be truncated to 2001:db8::/32.

    • description: (Optional string)

      A description of this ip range.

  • node_config: (Optional object(node_config))

    The configuration used for the Kubernetes Engine cluster.

    The node_config object accepts the following attributes:

    • zone: (Optional string)

      The Compute Engine zone in which to deploy the VMs running the Apache Airflow software, specified as the zone name or relative resource name (e.g. "projects/{project}/zones/{zone}"). Must belong to the enclosing environment's project and region. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • machine_type: (Optional string)

      The Compute Engine machine type used for cluster instances, specified as a name or relative resource name. For example: "projects/{project}/zones/{zone}/machineTypes/{machineType}". Must belong to the enclosing environment's project and region/zone. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • network: (Optional string)

      he Compute Engine network to be used for machine communications, specified as a self-link, relative resource name (for example "projects/{project}/global/networks/{network}"), by name.

      The network must belong to the environment's project. If unspecified, the "default" network ID in the environment's project is used. If a Custom Subnet Network is provided, subnetwork must also be provided.

    • subnetwork: (Optional string)

      The Compute Engine subnetwork to be used for machine communications, specified as a self-link, relative resource name (for example, "projects/{project}/regions/{region}/subnetworks/{subnetwork}"), or by name. If subnetwork is provided, network must also be provided and the subnetwork must belong to the enclosing environment's project and region.

    • disk_size_gb: (Optional number)

      The disk size in GB used for node VMs. Minimum size is 20GB. If unspecified, defaults to 100GB. Cannot be updated. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • oauth_scopes: (Optional set(string))

      The set of Google API scopes to be made available on all node VMs. Cannot be updated. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

      Default is ["https://www.googleapis.com/auth/cloud-platform"].

    • service_account: (Optional string)

      The Google Cloud Platform Service Account to be used by the node VMs. If a service account is not specified, the "default" Compute Engine service account is used. Cannot be updated. If given, note that the service account must have roles/composer.worker for any GCP resources created under the Cloud Composer Environment.

    • tags: (Optional set(string))

      The list of instance tags applied to all node VMs. Tags are used to identify valid sources or targets for network firewalls. Each tag within the list must comply with RFC1035. Cannot be updated. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • ip_allocation_policy: (Optional object(ip_allocation_policy))

      Configuration for controlling how IPs are allocated in the GKE cluster.

      The ip_allocation_policy object accepts the following attributes:

      • use_ip_aliases: (Optional bool)

        Whether or not to enable Alias IPs in the GKE cluster. If true, a VPC-native cluster is created. Defaults to true if the ip_allocation_policy block is present in config.

      • cluster_secondary_range_name: (Optional string)

        The name of the cluster's secondary range used to allocate IP addresses to pods. Specify either cluster_secondary_range_name or cluster_ipv4_cidr_block but not both. For Cloud Composer 1 environments, this field is applicable only when use_ip_aliases is true.

      • services_secondary_range_name: (Optional string)

        The name of the services' secondary range used to allocate IP addresses to the cluster. Specify either services_secondary_range_name or services_ipv4_cidr_block but not both. For Cloud Composer 1 environments, this field is applicable only when use_ip_aliases is true.

      • cluster_ipv4_cidr_block: (Optional string)

        The IP address range used to allocate IP addresses to pods in the cluster. For Cloud Composer 1 environments, this field is applicable only when use_ip_aliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either cluster_secondary_range_name or cluster_ipv4_cidr_block but not both.

      • cluster_ipv4_cidr_block: (Optional string)

        The IP address range used to allocate IP addresses in this cluster. For Cloud Composer 1 environments, this field is applicable only when use_ip_aliases is true. Set to blank to have GKE choose a range with the default size. Set to /netmask (e.g. /14) to have GKE choose a range with a specific netmask. Set to a CIDR notation (e.g. 10.96.0.0/14) from the RFC-1918 private networks (e.g. 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) to pick a specific range to use. Specify either services_secondary_range_name or services_ipv4_cidr_block but not both.

    • max_pods_per_node: (Optional number)

      The maximum pods per node in the GKE cluster allocated during environment creation. Lowering this value reduces IP address consumption by the Cloud Composer Kubernetes cluster. This value can only be set during environment creation, and only if the environment is VPC-Native. The range of possible values is 8-110, and the default is 32. Cannot be updated. This field is supported for Cloud Composer environments in versions composer-1.*.*-airflow-*.*.*.

    • enable_ip_masq_agent: (Optional bool)

      Deploys 'ip-masq-agent' daemon set in the GKE cluster and defines nonMasqueradeCIDRs equals to pod IP range so IP masquerading is used for all destination addresses, except between pods traffic. See: https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

  • kms_key_name: (Required string)

    Customer-managed Encryption Key available through Google's Key Management Service. It must be the fully qualified resource name, i.e. projects/project-id/locations/location/keyRings/keyring/cryptoKeys/key. Cannot be updated.

  • maintenance_window: (Optional object(maintenance_window))

    The configuration settings for Cloud Composer maintenance windows.

    The maintenance_window object accepts the following attributes:

    • start_time: (Required string)

      Start time of the first recurrence of the maintenance window

    • end_time: (Required string)

      Maintenance window end time. It is used only to calculate the duration of the maintenance window. The value for end-time must be in the future, relative to 'start_time'.

    • recurrence: (Required string)

      Maintenance window recurrence. Format is a subset of RFC-5545 (https://tools.ietf.org/html/rfc5545) 'RRULE'. The only allowed values for 'FREQ' field are 'FREQ=DAILY' and 'FREQ=WEEKLY;BYDAY=…'. Example values: 'FREQ=WEEKLY;BYDAY=TU,WE', 'FREQ=DAILY'.

Module Configuration

  • module_enabled: (Optional bool)

    Specifies whether resources in the module will be created.

    Default is true.

  • module_depends_on: (Optional list(dependency))

    A list of dependencies. Any object can be assigned to this list to define a hidden external dependency.

    Default is [].

    Example:

    module_depends_on = [
  null_resource.name
]

External Documentation

Google Documentation

Terraform AWS Provider Documentation

Module Versioning

This Module follows the principles of Semantic Versioning (SemVer).

Given a version number MAJOR.MINOR.PATCH, we increment the:

  1. MAJOR version when we make incompatible changes,
  2. MINOR version when we add functionality in a backwards compatible manner, and
  3. PATCH version when we make backwards compatible bug fixes.

Backwards compatibility in 0.0.z and 0.y.z version

  • Backwards compatibility in versions 0.0.z is not guaranteed when z is increased. (Initial development)
  • Backwards compatibility in versions 0.y.z is not guaranteed when y is increased. (Pre-release)

About Mineiros

Mineiros is a remote-first company headquartered in Berlin, Germany that solves development, automation and security challenges in cloud infrastructure.

Our vision is to massively reduce time and overhead for teams to manage and deploy production-grade and secure cloud infrastructure.

We offer commercial support for all of our modules and encourage you to reach out if you have any questions or need help. Feel free to email us at hello@mineiros.io or join our Community Slack channel.

Reporting Issues

We use GitHub Issues to track community reported issues and missing features.

Contributing

Contributions are always encouraged and welcome! For the process of accepting changes, we use Pull Requests. If you'd like more information, please see our Contribution Guidelines.

Makefile Targets

This repository comes with a handy Makefile. Run make help to see details on each available target.

License

license

This module is licensed under the Apache License Version 2.0, January 2004. Please see LICENSE for full details.

Copyright © 2020-2022 Mineiros GmbH

About

A Terraform module to create and manage Cloud Composer, a fully managed workflow orchestration service built on Apache Airflow, in Google Cloud https://cloud.google.com/composer

mineirosiacterraform-modulesterraform

License:Apache License 2.0

Languages

Language:HCL 64.4%Language:Makefile 21.0%Language:Go 14.6%