Giters

Rookout / terraform-aws-rookout-deployment

A Terraform module for deployment of Rookout on-premise controller and datastore on AWS cloud.

Home Page:https://registry.terraform.io/modules/rookout/rookout-deployment/aws/latest

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Deploy Rookout on AWS ECS Fargate Cluster using Terraform

This terraform depolying Rookout Controller and Rookout Datastore on AWS ECS Fargate cluster.

The module implements the following architecture (default deployment):

Network architecture (default deployment):

Prerequisites

  1. Terraform installed.
  2. AWS account inlcuding: AWS CLI installed.
    • The AWS default profile should be set with an access key and secret (reference).
    • Set profile if used non default profile. Run: export AWS_PROFILE="<profile_name>"
  3. Create a provider.tf file (reference).
  4. Get your organizational Rookout token, and pass it as a variable to this module 
    rookout_token = "..."

Components

This module deploy the Rookout ETL Controller by default. It also allows deployment of the Rookout Datastore, and a demo application with the Rookout agent.

The components to deploy can be configured with the folloiwng boolean variables:

    deploy_datastore = true/false
    deploy_demo_app = true/false

Certificate and DNS records management types

There are two methods for certificates and DNS record management that will change the network architecture.

Module-managed certificate and CNAME record

For deployments where domain_name is provided, a rookout.YOURDOMAIN subdomain will be created in a route53 public hosted zone, and associated by creating an NS record in your domain's public hosted zone. The subdomain will be used for the controller, datastore (optional) and demo application (optional). A certificate for this subdomain will be created in ACM. ALBs will also be created for those components. The created certificate and DNS records will be associated to those ALBs' domain names.

Note: If you don't use route53 as your DNS registry provider, please contact us.

For this type of deployment, provide the following variable:

    domain_name = "YOUR_DOMAIN"

The internal_controller_alb boolean variable (false by default) can be used to make the communication with the ETL Controller internal.

Self-managed certificate and CNAME record

For self managed certificate deployments CNAME record should be created for the Datastore and/or Controller, so we have two options.

  1. Provided ACM certificate for the Datastore (example)

This deployment will use the pre-imported ARN of the certificate in ACM (Body, private key, and chain of certificate are needed). THe certificate will be used by the datastore, therefore a CNAME record of the certificate's domain should be recored at your DNS provider with the Datstore endpoint (output of the module). A Controller will be deployed with an internal load balancer and can be reached from the VPC with the Controller's endpoint (output of the module).

    datastore_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"
  1. Provided ACM certificate for Datastore and Controller (for internet-facing controller) (example)

Same as the previous option, but, the controller is internet facing too. Same procedure of CNAME record registration should be preformed for the Datastore and Controller endpoints that match to their certificate's domain.

    datastore_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"
    controller_acm_certificate_arn = "PRE_IMPORTED_ACM_CERTIFICATE_ARN"

If a demo application is deployed, it should be configured with the Controller's CNAME record:

    demo_app_controller_host = "YOUR_CONTROLLER_CNAME_RECORD"

Deplyment matrixes

The following matrixes demonsrate application's components network mode by key variables. HTTP means internal traffic with ALB, TLS means external secured traffic with ALB. Demo means demo flask application provided with this module.

Endpoints

controller.PROVIDED_DOMAIN - url of the controller, used for SDK (rooks) when DNS provided.

datastore.PROVIDED_DOMAIN - url to the datastore, used with rookout client (web browser application) when DNS provided.

demo.PROVIDE_DOMAIN - flask demo application for debuging when DNS provided.

Advanced configuration

  • Provided Domain + VPC and subnets (example)

    You can configure the module to use an existing VPC (where your application is running) using the following variables:

        vpc_id = "<your's existing vpc id>"
    vpc_public_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
    vpc_private_subnets = ["<first_sub_domain>", "<second_sub_domain>"]

  • Provided Domain + VPC and subnets + ECS cluster (example)

    You can configure the module to use an existing VPC and ECS cluster (where your application is running) using the following variables:

        vpc_id = "<your's vpc id>"
    vpc_public_subnets = ["<first_sub_domain>", "<second_sub_domain>"]
    vpc_private_subnets = ["<first_sub_domain>", "<second_sub_domain>"]

    create_cluster = false
    cluster_name = "<your's existing cluster name>"

  • custom_iam_task_exec_role_arn - This variable can be used to overwrite the existing IAM Role of ECS tasks execution

  • deploy_alb - This variable can be set to false to disable the deployment of ALBs. If disabled, DNS subdomain and ACM certificate will not be generated. In this case, the following configuration should be set:

    deploy_alb = false
controller_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX"
datastore_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX" #if deploy_datastore=true
demo_app_target_group_arn = "arn:aws:elasticloadbalancing:AWS_REGION:ACCOUNT_ID:ARN_SUFFIX" #if deploy_demo_app=true

    If target groups are not passed, the loadbalancer configuration block in task defenitaion will be disbaled.

  • internal - boolean variable wich switches the ALBs to be internal only. if provided domain_name will create private hosted zone us that domain. Usually used with wildcard certificate.

  • wildcard certificate can be used with datastore_acm_certificate_arn and controller_acm_certificate_arn variables. when those used, please create CNAME records for controller and datastore out of outputs of this module (controller_endpoint, datastore_endpoint) to match the certificate's domain.

Requirements

Name Version
terraform >= 0.14
aws >= 5.0
awsutils >= 0.11.0

Providers

Name Version
aws 5.1.0

Modules

Name Source Version
acm terraform-aws-modules/acm/aws ~> 3.0
vpc terraform-aws-modules/vpc/aws 4.0.2

Resources

Name Type
aws_alb.controller resource
aws_alb.datastore resource
aws_alb.demo resource
aws_cloudwatch_log_group.demo resource
aws_cloudwatch_log_group.rookout resource
aws_cloudwatch_log_stream.controller_log_stream resource
aws_cloudwatch_log_stream.datastore_log_stream resource
aws_cloudwatch_log_stream.demo_log_stream resource
aws_ecs_cluster.rookout resource
aws_ecs_service.controller resource
aws_ecs_service.datastore resource
aws_ecs_service.demo resource
aws_ecs_task_definition.controller resource
aws_ecs_task_definition.datastore resource
aws_ecs_task_definition.demo resource
aws_iam_policy.task_exec_role resource
aws_iam_role.task_exec_role resource
aws_lb_listener.controller resource
aws_lb_listener.datastore resource
aws_lb_listener.demo resource
aws_lb_target_group.controller resource
aws_lb_target_group.datastore resource
aws_lb_target_group.demo resource
aws_route53_record.controller resource
aws_route53_record.datastore resource
aws_route53_record.demo resource
aws_route53_record.rookout resource
aws_route53_zone.sub_domain resource
aws_security_group.alb_controller resource
aws_security_group.alb_datastore resource
aws_security_group.alb_demo resource
aws_security_group.allow_demo resource
aws_security_group.controller resource
aws_security_group.datastore resource
aws_ecs_cluster.provided data source
aws_region.current data source
aws_route53_zone.selected data source

Inputs

Name Description Type Default Required
additional_controller_env_vars Additional env variables of contorller, configure as map of key=values any {} no
additional_datastore_env_vars Additional env variables of contorller, configure as map of key=values any {} no
additional_demo_app_env_vars Additional env variables of contorller, configure as map of key=values any {} no
cluster_name ECS cluster name, if we want to deploy to existing one string "" no
controller_acm_certificate_arn ARN of pre-imported SSL certificate to ACM for Rookouts controller public access, if datastore ACM provided controller alb will be internal string "" no
controller_alb_sg_igress_cidr_blocks Ingress CIDRs for controller's ALB security group list(string) 
[
  "0.0.0.0/0"
]
 no
controller_image Controller image name string "rookout/controller" no
controller_replicas Rookout's onprem controller replica count string 2 no
controller_resource Rookout's onprem controller resource map map(any) 
{
  "cpu": 2048,
  "memory": 4096
}
 no
controller_sg_igress_cidr_blocks Ingress CIDRs of controller security group list(string) 
[
  "0.0.0.0/0"
]
 no
controller_target_group_arn Target group used by controller ECS tasks string "" no
controller_version Controller image version string "latest" no
create_cluster whether create a cluster or use existing one bool true no
create_vpc # VPC variables. bool true no
custom_iam_task_exec_role_arn ECS execution IAM Role overwrite, please pass arn of existing IAM Role string "" no
datastore_acm_certificate_arn ARN of pre-imported SSL certificate to ACM for Rookouts datastore public access string "" no
datastore_alb_sg_igress_cidr_blocks Ingress CIDRs datastore's ALB security group list(string) 
[
  "0.0.0.0/0"
]
 no
datastore_image Datastore image name string "rookout/data-on-prem" no
datastore_resource Rookout's onprem datastore resource map map(any) 
{
  "cpu": 2048,
  "memory": 4096
}
 no
datastore_sg_igress_cidr_blocks Ingress CIDRs of datastore security group list(string) 
[
  "0.0.0.0/0"
]
 no
datastore_target_group_arn Target group used by datastore ECS tasks string "" no
datastore_version Datastore image version string "latest" no
demo_app_alb_sg_igress_cidr_blocks Ingress CIDRs datastore's ALB security group list(string) 
[
  "0.0.0.0/0"
]
 no
demo_app_controller_host Host which the demo rook connect to controller using WebSocket string "" no
demo_app_sg_igress_cidr_blocks Ingress CIDRs of datastore security group list(string) 
[
  "0.0.0.0/0"
]
 no
demo_app_target_group_arn Target group used by demo applicatino ECS tasks string "" no
deploy_alb Radio button to not deploy ALB for ECS tasks, if false please provide target group for each bool true no
deploy_datastore (Optional) If true will deploy demo Rookout's datastore locally bool true no
deploy_demo_app (Optional) If true will deploy demo flask application to start debuging bool false no
domain_name DNS domain which sub string "" no
enforce_token Whether to enforce the token in controller bool true no
environment Environment name string "rookout" no
internal Flag to switch the deployment to be internal bool false no
internal_controller_alb If domain provided, switching in on will make controller be reachable internaly only bool false no
region AWS region, using providers region as default string "" no
rookout_token Rookout token string "" no
subdomain_vpc_association flag for association of a hosted zone with a vpc bool true no
vpc_availability_zones n/a list(string) 
[
  ""
]
 no
vpc_cidr n/a string "172.30.1.0/25" no
vpc_id VPC id should be passed only if create_vpc = false string "" no
vpc_private_subnets n/a list(string) 
[
  "172.30.1.0/27",
  "172.30.1.32/27"
]
 no
vpc_public_subnets n/a list(string) 
[
  "172.30.1.64/27",
  "172.30.1.96/27"
]
 no

Outputs

Name Description
controller_dns Rookout's on-prem controller dns
controller_endpoint Rookout's on-prem controller endpoint
datastore_dns Rookout's on-prem datastore DNS
datastore_endpoint Rookout's on-prem datastore endpoint
demo_dns Rookout's flask application DNS
demo_endpoint Rookout's flask application endpoint
ecs_cluster_id ECS cluster
vpc_id VPC id that created

About

A Terraform module for deployment of Rookout on-premise controller and datastore on AWS cloud.

https://registry.terraform.io/modules/rookout/rookout-deployment/aws/latest

License:MIT License

Languages

Language:HCL 79.3%Language:Smarty 16.7%Language:Go 4.1%