AWS Msk Kafka Cluster

Terraform module which creates Msk Kafka Cluster on AWS.

These types of resources are supported:

Features

This module create a fully featured Msk Kafka Cluster on Aws. You could configure monitoring, encryption, server options, etc...

Usage

module "msk-cluster" {
  source  = "angelabad/msk-cluster/aws"

  cluster_name    = "kafka"
  instance_type   = "kafka.t3.small"
  number_of_nodes = 2
  client_subnets  = ["subnet-0ab97cbe1bd1406c2", "subnet-0d6cbf60360dbac64"]
  kafka_version   = "2.6.2"

  extra_security_groups = ["sg-019fc0f7d26f6c70f"]

  enhanced_monitoring = "PER_BROKER"

  s3_logs_bucket = aws_s3_bucket.logs.id
  s3_logs_prefix = "msklogs"

  prometheus_jmx_exporter  = true
  prometheus_node_exporter = true

  server_properties = {
    "auto.create.topics.enable"  = "true"
    "default.replication.factor" = "2"
  }

  encryption_in_transit_client_broker = "TLS"

  tags = {
    Owner       = "user"
    Environment = "dev"
  }
}

Requirements

Name	Version
terraform	>= 0.15
aws	>= 4.16
random	>= 2.1

Providers

Name	Version
aws	>= 4.16
random	>= 2.1

Modules

No modules.

Resources

Name	Type
aws_msk_cluster.this	resource
aws_msk_configuration.this	resource
aws_msk_scram_secret_association.this	resource
aws_security_group.this	resource
aws_security_group_rule.jmx-exporter	resource
aws_security_group_rule.msk-plain	resource
aws_security_group_rule.msk-tls	resource
aws_security_group_rule.node_exporter	resource
aws_security_group_rule.zookeeper-plain	resource
aws_security_group_rule.zookeeper-tls	resource
random_id.configuration	resource
aws_subnet.this	data source

Inputs

Name	Description	Type	Default	Required
client_authentication_sasl_iam_enabled	Enables IAM client authentication.	`bool`	`false`	no
client_authentication_sasl_scram_secrets_arns	Associates SCRAM secrets stored in the Secrets Manager. You need secret policy.	`list(string)`	`[]`	no
client_authentication_tls_certificate_authority_arns	List of ACM Certificate Authority Amazon Resource Names (ARNs).	`list(string)`	`[]`	no
client_authentication_unauthenticated_enabled	Enables unauthenticated access.	`bool`	`false`	no
client_subnets	A list of subnets to connect to in client VPC	`list(string)`	n/a	yes
cloudwatch_logs_group	Name of the Cloudwatch Log Group to deliver logs to.	`string`	`""`	no
cluster_name	Name of the MSK cluster.	`string`	n/a	yes
encryption_at_rest_kms_key_arn	You may specify a KMS key short ID or ARN (it will always output an ARN) to use for encrypting your data at rest. If no key is specified, an AWS managed KMS ('aws/msk' managed service) key will be used for encrypting the data at rest.	`string`	`""`	no
encryption_in_transit_client_broker	Encryption setting for data in transit between clients and brokers. Valid values: TLS, TLS_PLAINTEXT, and PLAINTEXT. Default value is TLS_PLAINTEXT.	`string`	`"TLS_PLAINTEXT"`	no
encryption_in_transit_in_cluster	Whether data communication among broker nodes is encrypted. Default value: true.	`bool`	`true`	no
enhanced_monitoring	Specify the desired enhanced MSK CloudWatch monitoring level to one of three monitoring levels: DEFAULT, PER_BROKER, PER_TOPIC_PER_BROKER or PER_TOPIC_PER_PARTITION. See Monitoring Amazon MSK with Amazon CloudWatch.	`string`	`"DEFAULT"`	no
extra_security_groups	A list of extra security groups to associate with the elastic network interfaces to control who can communicate with the cluster.	`list(string)`	`[]`	no
firehose_logs_delivery_stream	Name of the Kinesis Data Firehose delivery stream to deliver logs to.	`string`	`""`	no
instance_type	Specify the instance type to use for the kafka brokers. e.g. kafka.m5.large.	`string`	n/a	yes
kafka_version	Specify the desired Kafka software version.	`string`	n/a	yes
number_of_nodes	The desired total number of broker nodes in the kafka cluster. It must be a multiple of the number of specified client subnets.	`number`	n/a	yes
prometheus_jmx_exporter	Indicates whether you want to enable or disable the JMX Exporter.	`bool`	`false`	no
prometheus_node_exporter	Indicates whether you want to enable or disable the Node Exporter.	`bool`	`false`	no
provisioned_volume_throughput	Throughput value of the EBS volumes for the data drive on each kafka broker node in MiB per second. The minimum value is 250. The maximum value varies between broker type. See [https://docs.aws.amazon.com/msk/latest/developerguide/msk-provision-throughput.html#throughput-bottlenecks](documentation on throughput bottlenecks).	`number`	`null`	no
s3_logs_bucket	Name of the S3 bucket to deliver logs to.	`string`	`""`	no
s3_logs_prefix	Prefix to append to the folder name.	`string`	`""`	no
server_properties	A map of the contents of the server.properties file. Supported properties are documented in the MSK Developer Guide.	`map(string)`	`{}`	no
tags	A mapping of tags to assign to the resource.	`map(string)`	`{}`	no
volume_size	The size in GiB of the EBS volume for the data drive on each broker node.	`number`	`1000`	no

Outputs

Name	Description
arn	Amazon Resource Name (ARN) of the MSK cluster.
bootstrap_brokers	A comma separated list of one or more hostname:port pairs of kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set o PLAINTEXT or TLS_PLAINTEXT.
bootstrap_brokers_tls	A comma separated list of one or more DNS names (or IPs) and TLS port pairs kafka brokers suitable to boostrap connectivity to the kafka cluster. Only contains value if client_broker encryption in transit is set to TLS_PLAINTEXT or TLS.
current_version	Current version of the MSK Cluster used for updates, e.g. K13V1IB3VIYZZH
default_security_group	Msk cluster default security group id.
encryption_at_rest_kms_key_arn	The ARN of the KMS key used for encryption at rest of the broker data volumes.
zookeeper_connect_string	A comma separated list of one or more hostname:port pairs to use to connect to the Apache Zookeeper cluster.

Authors

Module managed by Angel Abad

License

Apache 2 Licensed. See LICENSE for full details

oforce / terraform-aws-msk-cluster