astronomer / terraform-provider-astro

Astro Terraform Provider

Terraform Provider Astro

Requirements

• Terraform >= 1.7
• Go >= 1.21

Building The Provider

1. Clone the repository
2. Enter the repository directory
3. Build the provider using the following Makefile command:

make dep
make build

4. The provider binary will be available in the bin directory

Adding Dependencies

This provider uses Go modules. Please see the Go documentation for the most up-to-date information about using Go modules.

To add a new dependency github.com/author/dependency to your Terraform provider:

go get github.com/author/dependency
go mod tidy

Then commit the changes to go.mod and go.sum.

Using the provider

1. Create an API Token to use in the provider. We recommend creating an organization API token since it is the most flexible but the type of your API token will depend on your use case.
2. Create a main.tf file with the following content:

terraform {
  required_providers {
    astro = {
      source = "astronomer/astro"
    }
  }
}

provider "astro" {
  organization_id = "<cuid>"
}

# your terraform commands here

3. Run the following commands to apply the provider:

export ASTRO_API_TOKEN=<token>
terraform init # only needed the first time - initializes a working directory and downloads the necessary provider plugins and modules and setting up the backend for storing your infrastructure's state
terraform plan # creates a plan consisting of a set of changes that will make your resources match your configuration
terraform apply # performs a plan just like terraform plan does, but then actually carries out the planned changes to each resource using the relevant infrastructure provider's API

Developing the Provider

If you wish to work on the provider, you'll first need Go installed on your machine (see Requirements above).

To compile the provider, see Building The Provider.

To add example docs, add the correspond .tf files to the examples directory. These should be added for every new data source and resource.

To run terraform with the provider, create a .terraformrc file in your home directory (~) with the following content to override the provider installation with the local build:

provider_installation {
  dev_overrides {
    "registry.terraform.io/astronomer/astro" = "~/terraform-provider-astro/bin" # Your path to the provider binary
  }
  direct {}
}

Example main.tf file for development and testing data sources and resources

terraform {
  required_providers {
    astro = {
      source = "astronomer/astro"
    }
  }
}

# provider configuration
provider "astro" {
  organization_id = "<cuid>"
}

# get information on an existing workspace
data "astro_workspace" "example" {
  id = "<cuid>"
}

# output the workspace data to the terminal
output "data_workspace_example" {
  value = data.astro_workspace.example
}

# create a new workspace
resource "astro_workspace" "tf_workspace" {
  name                  = "my workspace"
  description           = "my first workspace"
  cicd_enforced_default = false
}

# output the newly created workspace resource to the terminal
output "terraform_workspace" {
  value = astro_workspace.tf_workspace
}

# create a new cluster resource
resource "astro_cluster" "tf_cluster" {
    type = "DEDICATED"
    name = "my first cluster"
    region = "us-east-1"
    cloud_provider = "AWS"
    vpc_subnet_range = "172.20.0.0/20"
    workspace_ids = [astro_workspace.tf_workspace.id, data.astro_workspace.example.id]
    timeouts = {
        create = "3h"
        update = "2h"
        delete = "20m"
    }
}

# create a new dedicated deployment resource in that cluster
resource "astro_deployment" "tf_dedicated_deployment" {
  name        = "my first dedicated deployment"
  description = ""
  cluster_id  = astro_cluster.tf_cluster.id
  type = "DEDICATED"
  contact_emails = ["[e-mail]"]
  default_task_pod_cpu = "0.25"
  default_task_pod_memory = "0.5Gi"
  executor = "KUBERNETES"
  is_cicd_enforced = true
  is_dag_deploy_enabled = true
  is_development_mode = false
  is_high_availability = true
  resource_quota_cpu = "10"
  resource_quota_memory = "20Gi"
  scheduler_size = "SMALL"
  workspace_id = astro_workspace.tf_workspace.id
  environment_variables = [{
      key = "key1"
      value = "value1"
      is_secret = false
  }]
}

# create a new standard deployment resource
resource "astro_deployment" "tf_standard_deployment" {
  name        = "my first standard deployment"
  description = ""
  type = "STANDARD"
  cloud_provider = "AWS"
  region = "us-east-1"
  contact_emails = []
  default_task_pod_cpu = "0.25"
  default_task_pod_memory = "0.5Gi"
  executor = "CELERY"
  is_cicd_enforced = true
  is_dag_deploy_enabled = true
  is_development_mode = false
  is_high_availability = false
  resource_quota_cpu = "10"
  resource_quota_memory = "20Gi"
  scheduler_size = "SMALL"
  workspace_id = astro_workspace.tf_workspace.id
  environment_variables = []
  worker_queues = [{
      name = "default"
      is_default = true
      astro_machine = "A5"
      max_worker_count = 10
      min_worker_count = 0
      worker_concurrency = 1
  }]
}

Testing

Unit tests can be run with make test.

Acceptance tests

Acceptance integration tests use a Terraform CLI binary to run real Terraform commands against the Astro API. The goal is to approximate using the provider with Terraform in production as closely as possible.

Using the terraform-plugin-testing framework, each resource.Test runs an acceptance test on a resource.

• ProtoV6ProviderFactories: map of the provider factories that the test suite will use to create the provider - just has the astronomer provider
• PreCheck: a function that runs before the test suite starts to check that all the required environment variables are set
• Steps: a list of terraform apply sequences that the test suite will run. Each step is a resource.TestStep that contains a Config and Check function.
  • Config: the Terraform configuration that the test will run (ie. the .tf file)
  • Check: function that will verify the state of the resources after the terraform apply command has run.

In order to run the full suite of Acceptance tests, run make testacc. You will also need to set all the environment variables described in internal/provider/provider_test_utils.go.

The acceptance tests will run against the Astronomer API and create/read/update/delete real resources.

Importing Existing Resources

The Astro Terraform Import Script is a tool designed to help you import existing Astro resources into your Terraform configuration. Currently, this script automates the process of generating Terraform import blocks and resource configurations for the following resources: workspaces, deployments, clusters, hybrid cluster workspace authorizations, API tokens, teams, team roles, and user roles.

To use the import script, download the terraform-provider-astro-import-script executable file from releases based on your OS and architecture and run it with the following command:

On Unix-based systems:

chmod +x terraform-provider-astro-import-script_<version-number>_<os>_<arc>

./terraform-provider-astro-import-script_<version-number>_<os>_<arc> [options]

On Windows:

.\terraform-provider-astro-import-script_<version-number>_<os>_<arc>.exe [options]

Options

• -resources: Comma-separated list of resources to import. Accepted values are workspace, deployment, cluster, api_token, team, team_roles, user_roles.
• -token: API token to authenticate with the Astro platform. If not provided, the script will attempt to use the ASTRO_API_TOKEN environment variable.
• -organizationId: Organization ID to import resources from.
• -runTerraformInit: Run terraform init after generating the import configuration. Used for initializing the Terraform state in our GitHub Actions.
• -help: Display help information.

Examples

1. Import workspaces and deployments:

   ./terraform-provider-astro-import-script_<version-number>_<os>_<arc> -resources workspace,deployment -token your_api_token -organizationId your_org_id
   

2. Import all supported resources and run Terraform init:

   ./terraform-provider-astro-import-script_<version-number>_<os>_<arc> -resources workspace,deployment,cluster,api_token,team,team_roles,user_roles -token your_api_token -organizationId your_org_id -runTerraformInit
   

3. Use a different API host (e.g., dev environment):

   ./terraform-provider-astro-import-script_<version-number>_<os>_<arc> -resources workspace -token your_api_token -organizationId your_org_id
   

Output

The script will generate two main files:

1. import.tf: Contains the Terraform import blocks for the specified resources.
2. generated.tf: Contains the Terraform resource configurations for the imported resources.

Notes

• Ensure you have the necessary permissions in your Astro organization to access the resources you're attempting to import.
• The generated Terraform configurations may require some manual adjustment to match your specific requirements or to resolve any conflicts.
• Always review the generated files before applying them to your Terraform state.

FAQ and Troubleshooting

Frequently Asked Questions

1. What resources can I manage with this Terraform provider?

   • Workspaces, deployments, clusters, hybrid cluster workspace authorizations, API tokens, teams, team roles, and user roles.

2. How do I authenticate with the Astro API?

   • Use an API token set as the ASTRO_API_TOKEN environment variable or add it to the provider configuration.

3. Can I import existing Astro resources into Terraform?

   • Yes, use the Astro Terraform Import Script to generate import blocks and resource configurations.

4. What Terraform versions are required?

   • Terraform >= 1.7.

5. How can I contribute to the provider's development?

   • Submit pull requests, report issues, or suggest improvements on the GitHub repository.

Troubleshooting

1. Issue: 401 Unauthorized error when running terraform plan or terraform apply

   Solution: Your API token may have expired. Update your ASTRO_API_TOKEN environment variable with a fresh token:

   export ASTRO_API_TOKEN=<your-new-token>
   

2. Issue: Import script fails to find resources

   Solution:

   • Ensure you have the correct permissions in your Astro organization.
   • Verify that your API token is valid and has the necessary scopes and permissions.
   • Double-check the organization ID provided to the script.

3. Issue: "Error: Invalid provider configuration" when initializing Terraform

   Solution: Ensure your .terraformrc file is correctly set up, especially if you're using a local build of the provider for development.

If you encounter any issues not listed here, please check the GitHub Issues page or open a new issue with details about your problem.