This Terraform deploys a stateless containerised sshd bastion service on AWS with IAM based authentication:
This module requires Terraform >/=1.2.0 Older versions were previously supported going back to Terraform 0.11.x with module version to ~> v4.0
N.B. If you are using a newer version of this module when you have an older version deployed, please review the changelog!
This plan provides socket-activated sshd-containers with one container instantiated per connection and destroyed on connection termination or else after 12 hours- to deter things like reverse tunnels etc. The host assumes an IAM role, inherited by the containers, allowing it to query IAM users and request their ssh public keys lodged with AWS.
It is essential to limit incoming service traffic to whitelisted ports. If you do not then internet background noise will exhaust the host resources and/ or lead to rate limiting from amazon on the IAM identity calls- resulting in denial of service.
It is possible to replace the components in userdata and the base AMI with components of your own choosing. The following describes deployment with all sections as provided by module defaults.
The actual call for public keys is made with a GO binary, which is built during host instance initial launch and made available via shared volume in the docker image. In use the Docker container queries AWS for users with ssh keys at runtime, creates local linux user accounts for them and handles their login. The users who may access the bastion service may be restricted to membership of a defined AWS IAM group which is not set up or managed by this plan. When the connection is closed the container exits. This means that users log in as themselves and manage their own ssh keys using the AWS web console or CLI. For any given session they will arrive in a vanilla Ubuntu container with passwordless sudo and can install whatever applications and frameworks might be required for that session. Because the IAM identity checking and user account population is done at container run time and the containers are called on demand, there is no delay between creating an account with a public ssh key on AWS and being able to access the bastion. If users have more than one ssh public key then their account will be set up so that any of them may be used- AWS allows up to 5 keys per user. Aside from the resources provided by AWS and remote public repositories this plan is entirely self contained. There is no reliance on registries, build chains etc. Although it is possible to deploy this module as is without doing so, the standard/default configuration requires either a nat gateway on the VPC or a public IP address, in order for the instance to retrieve packages and dependencies to build the binary and containerised service
You may find it more convenient to call it in your plan directly from the Terraform Community Module Registry
Ivan Mesic has kindly contributed an example use of this module creating a VPC and a bastion instance within it - see /examples/full-with-public-ip
examples/custom-outbound-security-group is for more specialist use cases demonstrating how to run the service on a different port with an external supplied security group for external ingress and egress. This would not be necessary for most users.
You can specify a custom base AMI to use for the service host if you wish with var.custom_ami_id.
Userdata has been divided into sections which are individually applicable. Each is a HEREDOC and may be excluded by assigning any non-empty value to the relevant section variable. The value given is used simply for a logic test and not passed into userdata. If you ignore all of these variables then historic/ default behaviour continues and everything is built on the host instance on first boot (allow 3 minutes on t2.medium).
The variables for these sections are:
-
custom_ssh_populate - any value excludes default ssh_populate script used on container launch from userdata
-
custom_authorized_keys_command - any value excludes default Go binary iam-authorized-keys built from source from userdata
-
custom_docker_setup - any value excludes default docker installation and container build from userdata
-
custom_systemd - any value excludes default systemd and hostname change from userdata
-
extra_user_data* - (optional, several variables) you may supply your own user data here - appended following above sections
If you exclude any section then you must replace it with equivalent functionality, either in your base AMI or extra_user_data* for a working service. Especially if you are not replacing all sections then be mindful that the systemd service expects docker to be installed and to be able to call the docker container as sshd_worker. The service container in turn references the ssh_populate script which calls iam-authorized-keys from a specific location.
You can supply a list of one or more security groups to attach to the host instance launch configuration within the module if you wish. This can be supplied together with or instead of a whitelisted range of CIDR blocks. Starting with release 8.1 it is possible to use this to also passlist egress ports exclusively if var.custom_outbound_security_group = true (default false). It may be useful in an enterprise setting to have security groups with rules managed separately from the bastion plan but of course if you do not assign either a suitable security group or whitelist then you may not be able to reach the service!
Starting with release 8.1 it is possible to assign a custom port for the containerised ssh bastion service, e.g. port 443. This may be useful for advanced users and must match security group ingress and egress- see examples/custom-outbound-security-group
Load Balancer health check port may be optionally set to either the containerised service port (by default port 22) or port 2222 (EC2 host sshd). Port 2222 is the default. If you are deploying a large number of bastion instances, all of them checking into the same parent account for IAM queries in response to load balancer health checks on port 22 causes IAM rate limiting from AWS. Using the modified EC2 host sshd of port 2222 avoids this issue, is recommended for larger deployments and is now default. The host sshd is set to port 2222 as part of the service setup so this healthcheck is not entirely invalid. Security group rules, target groups and load balancer listeners are conditionally created to support any combination of access/healthcheck on port 2222 or not.
The ability to assume a role to source IAM users from another account has been integrated with conditional logic. If you supply the ARN for a role for the bastion service to assume (typically in another account) ${var.assume_role_arn} then this plan will create an instance profile, role and policy along with each bastion to make use of it. A matching sample policy and trust relationship is given as an output from the plan to assist with application in the other account. If you do not supply this arn then this plan presumes IAM lookups in the same account and creates an appropriate instance profile, role and policies for each bastion in the same AWS account. 'Each bastion' here refers to a combination of environment, AWS account, AWS region and VPCID determined by deployment. This is a high availability service, but if you are making more than one independent deployment using this same module within such a combination then you can specify "service_name" to avoid resource collision.
If you are seeking a solution for ECS hosts then you are recommended to the Widdix project. This offers IAM authentication for local users with a range of features suitable for a long-lived stateful host built as an AMI or with configuration management tools.
This plan creates a network load balancer and autoscaling group with an optional DNS entry and an optional public IP for the service.
You can overwrite the suggested hostname entirely with
var.bastion_host_nameYou can instead customise just the last part of the hostname if you like with
bastion_vpc_nameBy default this is the vpc ID via the magic default value of 'vpc_id' with the format
name = "${var.environment_name}-${data.aws_region.current.name}-${var.vpc}-bastion-service.${var.dns_domain}"e.g.
module default:
dev-ap-northeast-1-vpc-1a23b456d7890-bastion-service.yourdomain.combut you can pass a custom string, or an empty value to omit this. e.g.
bastion_vpc_name = "compute"gives
dev-ap-northeast-1-compute-bastion-service.yourdomain.com
and
bastion_vpc_name = ""gives
dev-ap-northeast-1-bastion-service.yourdomain.com
In any event this ensures a consistent and obvious naming format for each combination of AWS account and region that does not collide if multiple vpcs are deployed per region.
The container shell prompt is set similarly but with a systemd incremented counter, e.g. for
aws_useryou might see
aws_user@demo-eu-west-1-vpc-06ef3237ac609eb4b-0:~$ and a subsequent container might have
aws_user@demo-eu-west-1-vpc-06ef3237ac609eb4b-1:~$ In the case that
bastion_vpc_name = ""the service container shell prompt is set similar to
you@dev-ap-northeast-1_3It is considered normal to see very highly incremented counters if the load balancer health checks are conducted on the service port. It is essential to limit incoming service traffic to whitelisted ports. If you do not then internet background noise will exhaust the host resources and/ or lead to rate limiting from amazon on the IAM identity calls- resulting in denial of service.
The host is set to run the latest patch release at deployment of Debian Bullseye - unless you specify a custom AMI. Debian was chosen originally because the socket activation requires systemd but Ubuntu 16.04 did not automatically set up DHCP for additional elastic network interfaces (see version 1 series).
The host sshd is available on port 2222 and uses standard ec2 ssh keying. The default login username for Debian AMI's is 'admin'. If you do not whitelist any access to this port directly from the outside world (plan default) then it may be convenient to access from a container during development, e.g. with
sudo apt install -y curl; ssh -p2222 admin@`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`Make sure that your agent forwarding is active before attempting this!
It is advised to deploy to production without ec2 keys to increase security.
If you are interested in specifying your own AMI then be aware that there are many subtle differences in systemd implementations between different versions, e.g. it is not possible to use Amazon Linux 2 because we need (from Systemd):
- RunTimeMaxSec to limit the service container lifetime. This was introduced with Systemd version 229 (feb 2016) whereas Amazon Linux 2 uses version 219 (Feb 2015) This is a critical requirement.
with thanks to michaelwittig and the Widdix project
IAM user names may be up to 64 characters long.
Linux user names may only be up to 32 characters long.
Allowed characters for IAM user names are:
alphanumeric, including the following common characters: plus (+), equal (=), comma (,), period (.), at (@), underscore (_), and hyphen (-).
Allowed characters for Linux user names are (POSIX ("Portable Operating System Interface for Unix") standard (IEEE Standard 1003.1 2008)):
alphanumeric, including the following common characters: period (.), underscore (_), and hyphen (-).
Therefore, characters that are allowed in IAM user names but not in Linux user names:
plus (+), equal (=), comma (,), at (@).
This solution will use the following mapping for those special characters in iam usernames when creating linux user accounts on the sshd_worker container:
+=>plus==>equal,=>comma@=>at
So for example if we have an iam user called test@+=,test (which uses all of the disputed characters)
this username would translate to testatplusequalcommatest and they would need to shell in, e.g. with
ssh testatplusequalcommatest@dev-eu-west-1-bastion-service.yourdomain.com
- They are logging on as themselves using an identity based on their AWS IAM identity
- They must manage their own ssh keys using the AWS interface(s), e.g. in the web console under IAM/Users/Security credentials and 'Upload SSH public key'.
- The ssh server key is set at container build time. This means that it will change whenever the bastion host is respawned
The following is referenced in "message of the day" on the container:
- They have an Ubuntu userland with passwordless sudo within the container, so they can install whatever they find useful for that session
- Every connection is given a newly instanced container, nothing persists to subsequent connections. Even if they make a second connection to the service from the same machine at the same time it will be a separate container.
- When they close their connection that container terminates and is removed
- If they leave their connection open then the host will kill the container after 12 hours
The sshd-worker container is launched with -v /dev/log:/dev/log This causes logging information to be recorded in the host systemd journal which is not directly accessible from the container. It is thus simple to see who logged in and when by interrogating the host (if you have access!), e.g.
journalctl | grep 'Accepted publickey'gives information such as
April 27 14:05:02 dev-eu-west-1-bastion-host sshd[7294]: Accepted publickey for aws_user from 192.168.168.0 port 65535 ssh2: RSA SHA256:*****************************
N.B. It appears that calling client IP addresses are no longer visible - see issue 45. The reason for this is unclear.
Starting with release 3.8 it is possible to use the output giving the name of the role created for the service and to append additional user data. This means that you can call this module from a plan specifying your preferred logging solution, e.g. AWS cloudwatch.
- ssh keys are called only at login- if an account or ssh public key is deleted from AWS whilst a user is logged in then that session will continue until otherwise terminated.
EC2 Host OS (debian) with:
- Systemd docker unit
- Systemd service template unit
- IAM Profile connected to EC2 host
- golang
- go binary compiled from code included in plan and supplied as user data - sourced from Fullscreen project
IAM Role
This and all of the following are prefixed with ${var.service_name} to ensure uniqueness. An appropriate set is created depending on whether or not an external role to assume is referenced for IAM identity checks.
- IAM role
- IAM policies
- IAM instance profile
Docker container 'sshd_worker' - built at host launch time using generic ubuntu image, we add awscli; sshd and sudo.
Go binary and forked to a companion repo.
The files in question on the host deploy thus:
/opt
├── golang
│ ├── bin
│ ├── pkg
│ └── src
├── iam_helper
│ ├── iam-authorized-keys-command
│ └── ssh_populate.sh
└── sshd_worker
└── Dockerfile
golangis the source and build directory for the go binaryiam-helperis made available as a read-only volume to the docker container as /opt.iam-authorized-keys-commandis the Go binary that gets the users and ssh public keys from aws - it is built during bastion deployment. Built directly on the host - we may look at a dockerised build in futuressh_populate.shis the container entry point and populates the local user accounts using the go binarysshd_worker/Dockerfileis obviously the docker build configuration. It uses Ubuntu 22.04 from the public Docker registry and installs additional public packages.
If you supply the ARN for an external role for the bastion service to assume ${var.assume_role_arn} then a matching sample policy and trust relationship is given as an output from the plan to assist with application in that other account for typical operation.
The DNS entry (if created) for the service is also displayed as an output of the format
name = "${var.environment_name}-${data.aws_region.current.name}-${var.vpc}-bastion-service.${var.dns_domain}"These have been generated with terraform-docs
| Name | Version |
|---|---|
| terraform | >= 1.2.0 |
| Name | Version |
|---|---|
| aws | 4.22.0 |
| cloudinit | 2.2.0 |
No modules.
| Name | Description | Type | Default | Required |
|---|---|---|---|---|
| asg_desired | Desired numbers of bastion-service hosts in ASG | string |
"1" |
no |
| asg_max | Max numbers of bastion-service hosts in ASG | string |
"2" |
no |
| asg_min | Min numbers of bastion-service hosts in ASG | string |
"1" |
no |
| assume_role_arn | arn for role to assume in separate identity account if used | string |
"" |
no |
| autoscaling_group_enabled_metrics | A list of CloudWatch metrics to collect on the autoscaling group. Permitted values include: GroupMinSize; GroupMaxSize; GroupDesiredCapacity; GroupInServiceInstances; GroupPendingInstances; GroupStandbyInstances; GroupTerminatingInstances; GroupTotalInstances | list(string) |
[] |
no |
| aws_profile | n/a | string |
"" |
no |
| aws_region | n/a | any |
n/a | yes |
| bastion_allowed_iam_group | Name IAM group, members of this group will be able to ssh into bastion instances if they have provided ssh key in their profile | string |
"" |
no |
| bastion_ebs_device_name | Name of bastion instance block device | string |
"/dev/sda1" |
no |
| bastion_ebs_size | Size of EBS attached to the bastion instance | number |
8 |
no |
| bastion_host_name | The hostname to give to the bastion instance | string |
"" |
no |
| bastion_instance_types | List of ec2 types for the bastion host, used by aws_launch_template (first from the list) and in aws_autoscaling_group | list |
[ |
no |
| bastion_service_host_key_name | AWS ssh key *.pem to be used for ssh access to the bastion service host | string |
"" |
no |
| bastion_service_port | Port for containerised ssh daemon | number |
22 |
no |
| bastion_vpc_name | define the last part of the hostname, by default this is the vpc ID with magic default value of 'vpc_id' but you can pass a custom string, or an empty value to omit this | string |
"vpc_id" |
no |
| cidr_blocks_whitelist_host | range(s) of incoming IP addresses to whitelist for the HOST | list(string) |
[] |
no |
| cidr_blocks_whitelist_service | range(s) of incoming IP addresses to whitelist for the SERVICE | list(string) |
[] |
no |
| container_ubuntu_version | ubuntu version to use for service container | string |
"22.04" |
no |
| custom_ami_id | id for custom ami if used | string |
"" |
no |
| custom_authorized_keys_command | any value excludes default Go binary iam-authorized-keys built from source from userdata | string |
"" |
no |
| custom_docker_setup | any value excludes default docker installation and container build from userdata | string |
"" |
no |
| custom_outbound_security_group | don't create default outgoing permissive security group rule - will only work with custom AMI or if security group supplied with ports 53(UDP); 80(TCP); 443(TCP) open for 0.0.0.0/0 egress | bool |
false |
no |
| custom_ssh_populate | any value excludes default ssh_populate script used on container launch from userdata | string |
"" |
no |
| custom_systemd | any value excludes default systemd and hostname change from userdata | string |
"" |
no |
| delete_network_interface_on_termination | if network interface created for bastion host should be deleted when instance in terminated. Setting propagated to aws_launch_template.network_interfaces.delete_on_termination | bool |
true |
no |
| dns_domain | The domain used for Route53 records | string |
"" |
no |
| environment_name | the name of the environment that we are deploying to, used in tagging. Overwritten if var.service_name and var.bastion_host_name values are changed | string |
"staging" |
no |
| extra_user_data_content | Extra user-data to add to the default built-in | string |
"" |
no |
| extra_user_data_content_type | What format is content in - eg 'text/cloud-config' or 'text/x-shellscript' | string |
"text/x-shellscript" |
no |
| extra_user_data_merge_type | Control how cloud-init merges user-data sections | string |
"str(append)" |
no |
| lb_healthcheck_port | TCP port to conduct lb target group healthchecks. Acceptable values are 2222 or the value defined for bastion_service_port |
string |
"2222" |
no |
| lb_healthy_threshold | Healthy threshold for lb target group | string |
"2" |
no |
| lb_interval | interval for lb target group health check | string |
"30" |
no |
| lb_is_internal | whether the lb will be internal | string |
false |
no |
| lb_unhealthy_threshold | Unhealthy threshold for lb target group | string |
"2" |
no |
| on_demand_base_capacity | allows a base level of on demand when using spot | number |
0 |
no |
| public_ip | Associate a public IP with the host instance when launching | bool |
false |
no |
| route53_fqdn | If creating a public DNS entry with this module then you may override the default constructed DNS entry by supplying a fully qualified domain name here which will be used verbatim | string |
"" |
no |
| route53_zone_id | Route53 zoneId | string |
"" |
no |
| security_groups_additional | additional security group IDs to attach to host instance | list(string) |
[] |
no |
| service_name | Unique name per vpc for associated resources- set to some non-default value for multiple deployments per vpc | string |
"bastion-service" |
no |
| subnets_asg | list of subnets for autoscaling group - availability zones must match subnets_lb | list(string) |
n/a | yes |
| subnets_lb | list of subnets for load balancer - availability zones must match subnets_asg | list(string) |
n/a | yes |
| tags | AWS tags that should be associated with created resources | map(string) |
{} |
no |
| vpc | ID for Virtual Private Cloud to apply security policy and deploy stack to | any |
n/a | yes |
| Name | Description |
|---|---|
| bastion_service_assume_role_name | role created for service host asg - if created with assume role |
| bastion_service_role_name | role created for service host asg - if created without assume role |
| bastion_sg_id | Security Group id of the bastion host |
| lb_arn | aws load balancer arn |
| lb_dns_name | aws load balancer dns |
| lb_zone_id | n/a |
| policy_example_for_parent_account_empty_if_not_used | You must apply an IAM policy with trust relationship identical or compatible with this in your other AWS account for IAM lookups to function there with STS:AssumeRole and allow users to login |
| service_dns_entry | dns-registered url for service and host |
| target_group_arn | aws load balancer target group arn |