terraform-aws-arc-eks

Overview

SourceFuse AWS Reference Architecture (ARC) Terraform module for managing EKS clusters on AWS.

For more information about this repository and its usage, please see Terraform AWS ARC EKS Module Usage Guide.

Create the following AWS resources in a single region

eks cluster with essential network, security and logging configuration
eks fargate profile
eks node group

Bootstrap the eks cluster with the following components (WIP, currently unavailable)

cluster autoscaler
metrics server
kubernetes dashboard
AWS secrets store CSI driver

Usage

See example for complete example

module "eks_cluster" {
  source               = "sourcefuse/arc-eks/aws"
  environment          = var.environment
  name                 = var.name
  namespace            = var.namespace
  desired_size         = var.desired_size
  instance_types       = var.instance_types
  kubernetes_namespace = var.kubernetes_namespace
  create_node_group    = true
  max_size             = var.max_size
  min_size             = var.min_size
  subnet_ids           = data.aws_subnets.private.ids
  region               = var.region
  //  route_53_zone                             = var.route_53_zone
  vpc_id                    = data.aws_vpc.vpc.id
  enabled                   = true
  kubernetes_version        = var.kubernetes_version
  apply_config_map_aws_auth = true
  kube_data_auth_enabled    = true
  kube_exec_auth_enabled    = true
  csi_driver_enabled        = var.csi_driver_enabled
  map_additional_iam_roles  = var.map_additional_iam_roles
  allowed_security_groups   = concat(data.aws_security_groups.eks_sg.ids, data.aws_security_groups.db_sg.ids)
}

Note: This module creates aws-auth configmap and initialize it with the roles passed via the variable map_additional_iam_roles . It also creates a role {cluster-name}-eks-admin which can be assumed to get admin access to the cluster. Beyond that, it does not update that config or track any changes made to it. Hence the state of that configuration has to be independently managed.

Requirements

Name	Version
terraform	>= 1.6.0
aws	>= 5.0.0
helm	2.12.1
kubernetes	= 2.24.0
null	>= 2.0
tls	>= 3.1.0, != 4.0.0

Providers

Name	Version
aws	5.34.0
helm	2.12.1
kubernetes	2.24.0

Modules

Name	Source	Version
eks_cluster	cloudposse/eks-cluster/aws	3.0.0
eks_fargate_profile	cloudposse/eks-fargate-profile/aws	1.3.0
eks_node_group	cloudposse/eks-node-group/aws	2.12.0
this	cloudposse/label/null	0.25.0

Resources

Name	Type
aws_iam_role.eks_admin	resource
helm_release.csi_driver	resource
kubernetes_namespace.default_namespace	resource
aws_caller_identity.source	data source
aws_iam_policy_document.eks_admin	data source
aws_iam_policy_document.eks_admin_assume_role	data source

Inputs

Name	Description	Type	Default	Required
additional_tag_map	Additional key-value pairs to add to each map in `tags_as_list_of_maps`. Not added to `tags` or `id`. This is for some rare cases where resources want additional configuration of tags and therefore take a list of maps with tag key, value, and additional configuration.	`map(string)`	`{}`	no
addons	Manages `aws_eks_addon` resources	list(object({ addon_name = string addon_version = optional(string, null) configuration_values = optional(string, null) resolve_conflicts_on_create = optional(string, null) resolve_conflicts_on_update = optional(string, null) service_account_role_arn = optional(string, null) create_timeout = optional(string, null) update_timeout = optional(string, null) delete_timeout = optional(string, null) }))	`[]`	no
admin_principal	list of arns of IAM users/roles to be allowed to assume the eks-admin role. Default behaviour it to allow all users in the same AWS account as the caller	`list(string)`	`null`	no
allowed_cidr_blocks	A list of IPv4 CIDRs to allow access to the cluster. The length of this list must be known at "plan" time.	`list(string)`	`[]`	no
allowed_security_groups	List of Security Group IDs to be allowed to connect to the EKS cluster	`list(string)`	`[]`	no
ami_image_id	AMI to use. Ignored if `launch_template_id` is supplied.	`list(string)`	`[]`	no
ami_release_version	EKS AMI version to use, e.g. For AL2 "1.16.13-20200821" or for bottlerocket "1.2.0-ccf1b754" (no "v") or for Windows "2023.02.14". For AL2, bottlerocket and Windows, it defaults to latest version for Kubernetes version.	`list(string)`	`[]`	no
ami_type	Type of Amazon Machine Image (AMI) associated with the EKS Node Group. Defaults to `AL2_x86_64`. Valid values: `AL2_x86_64, AL2_x86_64_GPU, AL2_ARM_64, CUSTOM, BOTTLEROCKET_ARM_64, BOTTLEROCKET_x86_64, BOTTLEROCKET_ARM_64_NVIDIA, BOTTLEROCKET_x86_64_NVIDIA, WINDOWS_CORE_2019_x86_64, WINDOWS_FULL_2019_x86_64, WINDOWS_CORE_2022_x86_64, WINDOWS_FULL_2022_x86_64`.	`string`	`"AL2_x86_64"`	no
apply_config_map_aws_auth	Whether to apply the ConfigMap to allow worker nodes to join the EKS cluster and allow additional users, accounts and roles to acces the cluster	`bool`	`true`	no
attributes	ID element. Additional attributes (e.g. `workers` or `cluster`) to add to `id`, in the order they appear in the list. New attributes are appended to the end of the list. The elements of the list are joined by the `delimiter` and treated as a single ID element.	`list(string)`	`[]`	no
capacity_type	Type of capacity associated with the EKS Node Group. Valid values: "ON_DEMAND", "SPOT", or `null`. Terraform will only perform drift detection if a configuration value is provided.	`string`	`null`	no
cluster_encryption_config_enabled	Set to `true` to enable Cluster Encryption Configuration	`bool`	`true`	no
cluster_encryption_config_kms_key_deletion_window_in_days	Cluster Encryption Config KMS Key Resource argument - key deletion windows in days post destruction	`number`	`10`	no
cluster_encryption_config_kms_key_enable_key_rotation	Cluster Encryption Config KMS Key Resource argument - enable kms key rotation	`bool`	`true`	no
cluster_encryption_config_kms_key_id	KMS Key ID to use for cluster encryption config	`string`	`""`	no
cluster_encryption_config_kms_key_policy	Cluster Encryption Config KMS Key Resource argument - key policy	`string`	`null`	no
cluster_encryption_config_resources	Cluster Encryption Config Resources to encrypt, e.g. ['secrets']	`list(any)`	[ "secrets" ]	no
cluster_log_retention_period	Number of days to retain cluster logs. Requires `enabled_cluster_log_types` to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html.	`number`	`0`	no
context	Single object for setting entire context at once. See description of individual variables for details. Leave string and numeric variables as `null` to use default value. Individual variable settings (non-null) override settings in context object, except for attributes, tags, and additional_tag_map, which are merged.	`any`	{ "additional_tag_map": {}, "attributes": [], "delimiter": null, "descriptor_formats": {}, "enabled": true, "environment": null, "id_length_limit": null, "label_key_case": null, "label_order": [], "label_value_case": null, "labels_as_tags": [ "unset" ], "name": null, "namespace": null, "regex_replace_chars": null, "stage": null, "tags": {}, "tenant": null }	no
create_fargate_profile	Whether to create EKS Fargate profile	`bool`	`false`	no
create_node_group	Whether to create EKS Node Group	`bool`	`false`	no
csi_driver_enabled	The Secrets Store CSI Driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume.	`bool`	`false`	no
delimiter	Delimiter to be used between ID elements. Defaults to `-` (hyphen). Set to `""` to use no delimiter at all.	`string`	`null`	no
descriptor_formats	Describe additional descriptors to be output in the `descriptors` output map. Map of maps. Keys are names of descriptors. Values are maps of the form `{<br> format = string<br> labels = list(string)<br>}` (Type is `any` so the map values can later be enhanced to provide additional options.) `format` is a Terraform format string to be passed to the `format()` function. `labels` is a list of labels, in order, to pass to `format()` function. Label values will be normalized before being passed to `format()` so they will be identical to how they appear in `id`. Default is `{}` (`descriptors` output will be empty).	`any`	`{}`	no
desired_size	Desired number of worker nodes	`number`	n/a	yes
enabled	Set to false to prevent the module from creating any resources	`bool`	`false`	no
enabled_cluster_log_types	A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [`api`, `audit`, `authenticator`, `controllerManager`, `scheduler`]	`list(string)`	[ "api", "audit", "authenticator", "controllerManager", "scheduler" ]	no
environment	ID element. Usually used for region e.g. 'uw2', 'us-west-2', OR role 'prod', 'staging', 'dev', 'UAT'	`string`	`null`	no
id_length_limit	Limit `id` to this many characters (minimum 6). Set to `0` for unlimited length. Set to `null` for keep the existing setting, which defaults to `0`. Does not affect `id_full`.	`number`	`null`	no
instance_types	Set of instance types associated with the EKS Node Group. Defaults to ["t3.medium"]. Terraform will only perform drift detection if a configuration value is provided	`list(string)`	n/a	yes
kube_data_auth_enabled	If `true`, use an `aws_eks_cluster_auth` data source to authenticate to the EKS cluster. Disabled by `kubeconfig_path_enabled` or `kube_exec_auth_enabled`.	`bool`	`true`	no
kube_exec_auth_enabled	If `true`, use the Kubernetes provider `exec` feature to execute `aws eks get-token` to authenticate to the EKS cluster. Disabled by `kubeconfig_path_enabled`, overrides `kube_data_auth_enabled`.	`bool`	`false`	no
kubernetes_labels	Key-value mapping of Kubernetes labels. Only labels that are applied with the EKS API are managed by this argument. Other Kubernetes labels applied to the EKS Node Group will not be managed	`map(string)`	`{}`	no
kubernetes_namespace	Kubernetes namespace for selection	`string`	n/a	yes
kubernetes_version	Desired Kubernetes master version. If you do not specify a value, the latest available version is used	`string`	`"1.25"`	no
label_key_case	Controls the letter case of the `tags` keys (label names) for tags generated by this module. Does not affect keys of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper`. Default value: `title`.	`string`	`null`	no
label_order	The order in which the labels (ID elements) appear in the `id`. Defaults to ["namespace", "environment", "stage", "name", "attributes"]. You can omit any of the 6 labels ("tenant" is the 6th), but at least one must be present.	`list(string)`	`null`	no
label_value_case	Controls the letter case of ID elements (labels) as included in `id`, set as tag values, and output by this module individually. Does not affect values of tags passed in via the `tags` input. Possible values: `lower`, `title`, `upper` and `none` (no transformation). Set this to `title` and set `delimiter` to `""` to yield Pascal Case IDs. Default value: `lower`.	`string`	`null`	no
labels_as_tags	Set of labels (ID elements) to include as tags in the `tags` output. Default is to include all labels. Tags with empty values will not be included in the `tags` output. Set to `[]` to suppress all generated tags. Notes: The value of the `name` tag, if included, will be the `id`, not the `name`. Unlike other `null-label` inputs, the initial setting of `labels_as_tags` cannot be changed in later chained modules. Attempts to change it will be silently ignored.	`set(string)`	[ "default" ]	no
launch_template_id	The ID (not name) of a custom launch template to use for the EKS node group. If provided, it must specify the AMI image ID.	`list(string)`	`[]`	no
launch_template_version	The version of the specified launch template to use. Defaults to latest version.	`list(string)`	`[]`	no
local_exec_interpreter	shell to use for local_exec	`list(string)`	[ "/bin/sh", "-c" ]	no
map_additional_iam_roles	Additional IAM roles to add to `config-map-aws-auth` ConfigMap	list(object({ rolearn = string username = string groups = list(string) }))	`[]`	no
map_additional_iam_users	Additional IAM users to add to `config-map-aws-auth` ConfigMap	list(object({ userarn = string username = string groups = list(string) }))	`[]`	no
max_size	The maximum size of the AutoScaling Group	`number`	n/a	yes
min_size	The minimum size of the AutoScaling Group	`number`	n/a	yes
name	ID element. Usually the component or solution name, e.g. 'app' or 'jenkins'. This is the only ID element not also included as a `tag`. The "name" tag is set to the full `id` string. There is no tag with the value of the `name` input.	`string`	`null`	no
namespace	ID element. Usually an abbreviation of your organization name, e.g. 'eg' or 'cp', to help ensure generated IDs are globally unique	`string`	`null`	no
oidc_provider_enabled	Create an IAM OIDC identity provider for the cluster, then you can create IAM roles to associate with a service account in the cluster, instead of using `kiam` or `kube2iam`. For more information, see https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html	`bool`	`true`	no
public_access_cidrs	Specify the cidr blocks which will be able to access the eks public api endpoint	`list(string)`	[ "0.0.0.0/0" ]	no
regex_replace_chars	Terraform regular expression (regex) string. Characters matching the regex will be removed from the ID elements. If not set, `"/[^a-zA-Z0-9-]/"` is used to remove all characters other than hyphens, letters and digits.	`string`	`null`	no
region	AWS region	`string`	n/a	yes
stage	ID element. Usually used to indicate role, e.g. 'prod', 'staging', 'source', 'build', 'test', 'deploy', 'release'	`string`	`null`	no
subnet_ids	Subnet IDs	`list(string)`	n/a	yes
tags	Additional tags (e.g. `{'BusinessUnit': 'XYZ'}`). Neither the tag keys nor the tag values will be modified by this module.	`map(string)`	`{}`	no
tenant	ID element _(Rarely used, not included by default)_. A customer identifier, indicating who this instance of a resource is for	`string`	`null`	no
vpc_id	VPC ID	`string`	n/a	yes

Outputs

Name	Description
eks_cluster_arn	The Amazon Resource Name (ARN) of the cluster
eks_cluster_ca_certificate	CA certificate
eks_cluster_endpoint	The endpoint for the Kubernetes API server
eks_cluster_id	The name of the cluster
eks_cluster_identity_oidc_issuer	The OIDC Identity issuer for the cluster
eks_cluster_managed_security_group_id	Security Group ID that was created by EKS for the cluster. EKS creates a Security Group and applies it to ENI that is attached to EKS Control Plane master nodes and to any managed workloads
eks_cluster_security_group_arn	ARN of the EKS cluster Security Group
eks_cluster_security_group_id	ID of the EKS cluster Security Group
eks_cluster_security_group_name	Name of the EKS cluster Security Group
eks_cluster_version	The Kubernetes server version of the cluster
eks_node_group_arn	Amazon Resource Name (ARN) of the EKS Node Group
eks_node_group_id	EKS Cluster name and EKS Node Group name separated by a colon
eks_node_group_resources	List of objects containing information about underlying resources of the EKS Node Group
eks_node_group_role_arn	ARN of the worker nodes IAM role
eks_node_group_role_name	Name of the worker nodes IAM role
eks_node_group_status	Status of the EKS Node Group
eks_oidc_issuer_arn	EKS Cluster OIDC issuer
iam_role_eks_admin_arn	ARN for eks-admin iam role. Assume this to get admin access to the cluster

Kubernetes dashboard

To view the dashboard, run the following commands:

aws eks update-kubeconfig --name refarch-dev-primary-k8s-cluster --region us-east-1
kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-dashboard-viewer | awk '{print $1}') #Copy the token from the output
kubectl proxy

Then navigate to:

http://localhost:8001/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login

and use the copied token to login

Development

Prerequisites

Configurations

Configure pre-commit hooks

pre-commit install

Configure golang deps for tests

> go get github.com/gruntwork-io/terratest/modules/terraform
> go get github.com/stretchr/testify/assert

Git commits

while Contributing or doing git commit please specify the breaking change in your commit message whether its major,minor or patch

For Example

git commit -m "your commit message #major"

By specifying this , it will bump the version and if you dont specify this in your commit message then by default it will consider patch and will bump that accordingly