EMR on EKS with Terraform

There's currently a pull request open for EMR on EKS support in Terraform.

Until that gets merged, you can build your own version of the AWS Provider to test out the new code.

💁 Please note that this code is a proof-of-concept and not intended for production use

Building the provider

Requirements

Terraform v0.14+ (to use the custom provider and deploy your cluster)
Go 1.16 (to build the provider plugin)
- Do NOT use Go 1.17 as it will error on gofmt
- Also if you can not use the offical Go module proxy, the project will not build due to this issue

Here's the full documentation on building the AWS provider, but in short here's what you need to do.

Clone the repository and change into the directory

git clone git@github.com:terraform-providers/terraform-provider-aws
cd terraform-provider-aws

Checkout the pull request.

git fetch origin pull/20003/head:emr-eks-terraform
git checkout emr-eks-terraform

Run make tools. This will install the needed tools for the provider.

make tools

To compile the provider, run make build. This will build the provider and put the provider binary in the $GOPATH/bin (default: $HOME/go) directory.

make build

You should now have a terraform-provider-aws binary in your $GOPATH

which terraform-provider-aws

You'll need this path for the next step.

Override your AWS provider

In order to make use of the change, you need to populate your Terraform CLI configuration file (~/.terraformrc) with the directory your new binary is in.

provider_installation {
  dev_overrides {
    "hashicorp/aws" = "/Users/username/go/bin"
  }
  direct {}
}

Deploy your EMR on EKS virtual cluster!

One note before getting started. The version of the EMR on EKS module needs to be pinned to 17.9.0 due to a breaking change and the fact this PR is based off a specific version of the AWS provider.

The Terraform example provisions everything you need:

A new VPC (with the AWS VPC module)
An EKS cluster with a managed node group (with the AWS EKS module)
All the necessary IAM and Kubernetes roles as mentioned in the EMR on EKS Getting Started documenation

Please note that I've only run this end-to-end a couple times, so there may be some dependencies that require you to run terraform apply again.

By default, Terraform will use your default AWS profile and region.

terraform init
terraform apply

The EKS cluster can take about 10-15 minutes to provision.

Running a job

Now that your cluster is up, you should be able to run a job. Because the EKS cluster autoscaler is not installed by default, if you want to anything more complicated than the example below, you'll need to scale the cluster up manually.

The below example just runs a sample calculation of Pi and enables CloudWatch logging.

The EMR on EKS virtual cluster ID is provided in a emr-virtual-cluster-id output and the role to run the job is in the emr-eks-job-role output.

export EMR_EKS_CLUSTER_ID=<CLUSTER_ID>
export EMR_EKS_EXECUTION_ARN=arn:aws:iam::<ACCOUNT_ID>:role/tf_emr_eks_job_role

aws emr-containers start-job-run \
    --virtual-cluster-id ${EMR_EKS_CLUSTER_ID} \
    --name sample-pi \
    --execution-role-arn ${EMR_EKS_EXECUTION_ARN} \
    --release-label emr-6.3.0-latest \
    --job-driver '{
        "sparkSubmitJobDriver": {
            "entryPoint": "local:///usr/lib/spark/examples/src/main/python/pi.py"
        }
    }' \
    --configuration-overrides '{
        "monitoringConfiguration": {
            "cloudWatchMonitoringConfiguration": {
                "logGroupName": "/aws/eks/emr-spark",
                "logStreamNamePrefix": "pi"
            }
        }
    }'

References

IAM/K8s Roles

irsa example: https://github.com/terraform-aws-modules/terraform-aws-eks/tree/a26c9fd0c9c880d5b99c438ad620e91dda957e10/examples/irsa StringLike condition?: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/irsa/irsa.tf#L8 And another StringLike example: https://github.com/cloudposse/terraform-aws-eks-iam-role/blob/master/main.tf#L55 helpful re: roles: hashicorp/terraform-provider-kubernetes#322

Issues

update_config was broken: terraform-aws-modules/terraform-aws-eks#1570

rbrto / emr-eks-terraform