Giters

cschubiner / terraform-aws-kubespot

Kubernetes (AWS EKS) Terraform Modules for HIPAA/PCI/SOC2 Compliance and Cloud Security

Home Page:https://www.opszero.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Kubespot (AWS)

Compliance Oriented Kubernetes Setup for AWS.

Kubespot is an open source terraform module that attempts to create a complete compliance-oriented Kubernetes setup on AWS, Google Cloud and Azure. These add additional security such as additional system logs, file system monitoring, hard disk encryption and access control. Further, we setup the managed Redis and SQL on each of the Cloud providers with limited access to the Kubernetes cluster so things are further locked down. All of this should lead to setting up a HIPAA / PCI / SOC2 being made straightforward and repeatable.

This covers how we setup your infrastructure on AWS, Google Cloud and Azure. These are the three Cloud Providers that we currently support to run Kubernetes. Further, we use the managed service provided by each of the Cloud Providers. This document covers everything related to how infrastructure is setup within each Cloud, how we create an isolated environment for Compliance and the commonalities between them.

Tools & Setup

brew install kubectl kubernetes-helm awscli terraform

Cluster Usage

If the infrastructure is using the opsZero infrastructure as code template then you access the resources like the following:

Add your IAM credentials in ~/.aws/credentials.

[profile_name]
aws_access_key_id=<>key>
aws_secret_access_key=<secret_key>
region=us-west-2

cd environments/<nameofenv>
make kubeconfig
export KUBECONFIG=./kubeconfig # add to a .zshrc
kubectl get pods

Autoscaler

Kubespot uses Karpenter as the default autoscaler. To configure the autoscaler we need to create a file like the one below and run:

kubectl apply -f karpenter.yml
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  consolidation:
    enabled: true # If set to true the nodes will minimize to fit the pods
  requirements:
    - key: "karpenter.k8s.aws/instance-category"
      operator: In
      values: ["t", "c", "m"]
    - key: "kubernetes.io/arch"
      operator: In
      values: ["amd64"]
    - key: "karpenter.k8s.aws/instance-cpu"
      operator: In
      values: ["1", "2", "4", "8", "16"]
    - key: "karpenter.k8s.aws/instance-hypervisor"
      operator: In
      values: ["nitro"]
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot", "on-demand"]
  limits:
    resources:
      cpu: 200
  provider:
    securityGroupSelector:
      Name: <cluster-name>-node
    subnetSelector:
      Name: <cluster-name>-private
    tags:
      karpenter.sh/discovery: <cluster-name>
  ttlSecondsUntilExpired: 86400 # How long to keep the node before cycling

Cluster Setup

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Pro Support

opsZero provides support for our modules including:

  • Email support
  • Zoom Calls
  • Implementation Guidance

Providers

Name Version
aws ~> 4.50.0
aws.virginia ~> 4.50.0
helm 2.8.0
http n/a
kubernetes >= 2.0
null n/a
tls n/a

Inputs

Name Description Type Default Required
alb_controller_version The chart version of the ALB controller helm chart string "1.4.4" no
aws_load_balancer_controller_enabled Enable ALB controller by default bool true no
aws_profile AWS profile to use string n/a yes
cidr_block The CIDR block used by the VPC string "10.2.0.0/16" no
cidr_block_private_subnet The CIDR block used by the private subnet list 
[
  "10.2.2.0/24",
  "10.2.3.0/24"
]
 no
cidr_block_public_subnet The CIDR block used by the private subnet list 
[
  "10.2.0.0/24",
  "10.2.1.0/24"
]
 no
cluster_logging List of the desired control plane logging to enable list 
[
  "api",
  "audit",
  "authenticator",
  "controllerManager",
  "scheduler"
]
 no
cluster_private_access Whether the Amazon EKS private API server endpoint is enabled bool false no
cluster_public_access Whether the Amazon EKS private API server endpoint is enabled bool true no
cluster_public_access_cidrs List of CIDR blocks. Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled list 
[
  "0.0.0.0/0"
]
 no
cluster_version Desired Kubernetes master version string "1.23" no
csi_secrets_store_enabled Specify whether the CSI driver is enabled on the EKS cluster bool false no
csi_secrets_store_version The version of the CSI store helm chart string "1.3.0" no
efs_enabled Specify whether the EFS is enabled on the EKS cluster bool false no
eips List of Elastic IPs list [] no
enable_egress_only_internet_gateway Create an egress-only Internet gateway for your VPC0 bool false no
enable_ipv6 Enable an Amazon-provided IPv6 CIDR block with a /56 prefix length for the VPC bool false no
enable_nat Whether the NAT gateway is enabled bool true no
enabled_metrics_asg A list of metrics to collect list 
[
  "GroupDesiredCapacity",
  "GroupInServiceCapacity",
  "GroupInServiceInstances",
  "GroupMaxSize",
  "GroupMinSize",
  "GroupPendingCapacity",
  "GroupPendingInstances",
  "GroupStandbyCapacity",
  "GroupStandbyInstances",
  "GroupTerminatingCapacity",
  "GroupTerminatingInstances",
  "GroupTotalCapacity",
  "GroupTotalInstances"
]
 no
environment_name Name of the environment to create AWS resources string n/a yes
fargate_selector Terraform object to create the EKS fargate profiles map 
{
  "serverless": {}
}
 no
govcloud Set if the environment is govcloud bool false no
iam_users List of IAM users list [] no
karpenter_enabled Specify whether the karpenter is enabled bool false no
karpenter_version The version of the karpenter helm chart string "v0.23.0" no
legacy_subnet Specify how the subnets should be created bool true no
metrics_server_version The version of the metric server helm chart string "3.8.3" no
monitoring_role_arn The ARN for the IAM role that permits RDS to send enhanced monitoring metrics to CloudWatch Logs string "" no
node_group_cpu_threshold The value of the CPU threshold string "70" no
node_groups Terraform object to create the EKS node groups map {} no
node_role_policies A list of The ARN of the policies you want to attach list [] no
nodes_blue_desired_capacity The number of Amazon EC2 instances that should be running in the group number 0 no
nodes_blue_instance_type The instance type to use for the instance string "t3.micro" no
nodes_blue_max_instance_lifetime The maximum amount of time, in seconds, that an instance can be in service number 604800 no
nodes_blue_max_size The maximum size of the Auto Scaling Group number 0 no
nodes_blue_min_size The minimum size of the Auto Scaling Group number 0 no
nodes_blue_root_device_size Size of the volume in gibibytes (GiB) string "20" no
nodes_blue_spot_price The maximum price to use for reserving spot instances. any null no
nodes_blue_subnet_ids A list of subnet IDs to launch resources in list [] no
nodes_green_desired_capacity The number of Amazon EC2 instances that should be running in the group number 0 no
nodes_green_instance_type The instance type to use for the instance string "t3.micro" no
nodes_green_max_instance_lifetime The maximum amount of time, in seconds, that an instance can be in service number 604800 no
nodes_green_max_size The maximum size of the Auto Scaling Group number 0 no
nodes_green_min_size The minimum size of the Auto Scaling Group number 0 no
nodes_green_root_device_size Size of the volume in gibibytes (GiB) string "20" no
nodes_green_spot_price The maximum price to use for reserving spot instances. any null no
nodes_green_subnet_ids A list of subnet IDs to launch resources in list [] no
nodes_in_public_subnet INSECURE! Only use this if you want to avoid paying for the NAT. Also set enable_nat to false bool false no
redis_enabled Whether the redis cluster is enabled bool false no
redis_engine_version Version number of the cache engine to be used for the cache clusters in this replication group string "7.0" no
redis_node_type Instance class of the redis cluster to be used string "cache.t4g.micro" no
redis_num_nodes Number of nodes for redis number 1 no
repos n/a list [] no
sql_cluster_enabled Whether the sql cluster is enabled bool false no
sql_database_name The name of the database to create when the DB instance is created string "" no
sql_encrypted Specify whether the DB instance is encrypted bool true no
sql_engine The name of the database engine to be used for this DB cluster string "aurora-postgresql" no
sql_engine_mode The database engine mode string "provisioned" no
sql_engine_version The engine version to use string "14.3" no
sql_identifier The name of the database string "" no
sql_instance_allocated_storage The allocated storage in gibibytes number 20 no
sql_instance_class The instance type of the RDS instance. string "db.t4g.micro" no
sql_instance_enabled Whether the sql instance is enabled bool false no
sql_instance_engine The database engine to use string "postgres" no
sql_instance_max_allocated_storage the upper limit to which Amazon RDS can automatically scale the storage of the DB instance number 200 no
sql_master_password Password for the master DB user string "" no
sql_master_username Username for the master DB user string "" no
sql_node_count The number of instances to be used for this DB cluster number 0 no
sql_parameter_group_name Name of the DB parameter group to associate string "" no
sql_rds_multi_az Specify if the RDS instance is enabled multi-AZ bool false no
sql_serverless_max The maximum capacity for the DB cluster number 2 no
sql_serverless_min The maximum capacity for the DB cluster number 2 no
sql_serverless_seconds_until_auto_pause The time, in seconds, before the DB cluster in serverless mode is paused number 300 no
sql_skip_final_snapshot Determines whether a final DB snapshot is created before the DB instance is deleted. bool true no
sql_subnet_group_include_public Include public subnets as part of the clusters subnet configuration. bool false no
sso_roles Terraform object of the IAM roles map 
{
  "admin_roles": [],
  "dev_roles": [],
  "monitoring_roles": [],
  "readonly_roles": []
}
 no
tags Terraform map to create custom tags for the AWS resources map {} no
vpc_flow_logs_enabled Specify whether the vpc flow log is enabled bool false no
zones AZs for the subnets list 
[
  "us-west-2a",
  "us-west-2b"
]
 no

Resources

Name Type
aws_autoscaling_group.nodes_blue resource
aws_autoscaling_group.nodes_green resource
aws_cloudwatch_log_group.vpc resource
aws_cloudwatch_metric_alarm.database_cpu_database resource
aws_cloudwatch_metric_alarm.database_disk_database resource
aws_cloudwatch_metric_alarm.database_free_disk_database resource
aws_cloudwatch_metric_alarm.database_free_disk_database2 resource
aws_cloudwatch_metric_alarm.database_free_disk_database3 resource
aws_cloudwatch_metric_alarm.database_free_disk_database4 resource
aws_cloudwatch_metric_alarm.database_io_mysql resource
aws_cloudwatch_metric_alarm.database_io_postgres resource
aws_cloudwatch_metric_alarm.node_group_cpu_threshold resource
aws_cloudwatch_metric_alarm.nodes_blue_cpu_threshold resource
aws_cloudwatch_metric_alarm.nodes_green_cpu_threshold resource
aws_db_instance.default resource
aws_db_subnet_group.default resource
aws_egress_only_internet_gateway.egress resource
aws_eip.eips resource
aws_eks_addon.core resource
aws_eks_cluster.cluster resource
aws_eks_fargate_profile.fargate resource
aws_eks_node_group.node_group resource
aws_elasticache_cluster.default resource
aws_elasticache_subnet_group.default resource
aws_flow_log.vpc resource
aws_iam_instance_profile.node resource
aws_iam_openid_connect_provider.cluster resource
aws_iam_policy.alb resource
aws_iam_policy.ebs resource
aws_iam_policy.efs_policy resource
aws_iam_role.cluster resource
aws_iam_role.fargate resource
aws_iam_role.node resource
aws_iam_role.vpc resource
aws_iam_role_policy.vpc resource
aws_iam_role_policy_attachment.alb resource
aws_iam_role_policy_attachment.cluster-AmazonEKSClusterPolicy resource
aws_iam_role_policy_attachment.cluster-AmazonEKSServicePolicy resource
aws_iam_role_policy_attachment.ebs resource
aws_iam_role_policy_attachment.fargate-AmazonEKSFargatePodExecutionRolePolicy resource
aws_iam_role_policy_attachment.node-AmazonEC2ContainerRegistryReadOnly resource
aws_iam_role_policy_attachment.node-AmazonEKSWorkerNodePolicy resource
aws_iam_role_policy_attachment.node-AmazonEKS_CNI_Policy resource
aws_iam_role_policy_attachment.node-EFS resource
aws_iam_role_policy_attachment.node_role_policies resource
aws_internet_gateway.public resource
aws_launch_configuration.nodes_blue resource
aws_launch_configuration.nodes_green resource
aws_nat_gateway.gw resource
aws_rds_cluster.default resource
aws_rds_cluster_instance.cluster_instances resource
aws_route.ig resource
aws_route.ipv6 resource
aws_route.nat resource
aws_route_table.private resource
aws_route_table.public resource
aws_route_table_association.private resource
aws_route_table_association.public resource
aws_security_group.cluster resource
aws_security_group.node resource
aws_security_group_rule.cluster-ingress-node-https resource
aws_security_group_rule.eks resource
aws_security_group_rule.node-ingress-cluster resource
aws_security_group_rule.node-ingress-self resource
aws_security_group_rule.private_subnet resource
aws_security_group_rule.public_subnet resource
aws_subnet.private resource
aws_subnet.public resource
aws_vpc.vpc resource
helm_release.aws_efs_csi_driver resource
helm_release.aws_load_balancer resource
helm_release.csi_secrets_store resource
helm_release.karpenter resource
helm_release.metrics-server resource
kubernetes_cluster_role_binding.eks_admins_binding resource
kubernetes_cluster_role_binding.eks_readonly_binding resource
kubernetes_config_map.aws_auth resource
kubernetes_role.default_eks_admins resource
kubernetes_role.default_eks_developers resource
kubernetes_role.default_eks_monitoring_admins resource
kubernetes_role.default_eks_readonly resource
kubernetes_role_binding.default_eks_admins resource
kubernetes_role_binding.default_eks_developers resource
kubernetes_role_binding.default_eks_monitoring_admins resource
kubernetes_role_binding.default_eks_readonly resource
null_resource.csi_secrets_store_aws_provider resource
null_resource.karpenter_awsnodetemplates_crd resource
null_resource.karpenter_crd resource
aws_availability_zones.available data source
aws_caller_identity.current data source
aws_ecrpublic_authorization_token.token data source
aws_eks_cluster_auth.cluster data source
aws_iam_policy.ssm_managed_instance data source
aws_partition.current data source
aws_region.current data source
aws_ssm_parameter.eks_ami data source
http_http.csi_secrets_store_aws_provider data source
http_http.karpenter_crd data source
tls_certificate.cluster data source

Outputs

Name Description
eks_cluster n/a
eks_cluster_token n/a
nat_gateway_ids n/a
node_role n/a
node_security_group_id n/a
private_route_table n/a
private_subnet_ids n/a
public_route_table n/a
public_subnet_ids n/a
redis_elasticache_subnet_group_name n/a
vpc_id n/a

About

Kubernetes (AWS EKS) Terraform Modules for HIPAA/PCI/SOC2 Compliance and Cloud Security

https://www.opszero.com

License:MIT License

Languages

Language:HCL 100.0%Language:Makefile 0.0%