A Table of Contents for All Gruntwork Code

This repo is a convenient way to find all of the repositories created and maintained by Gruntwork. It is intended primarily for Gruntwork's customers.

Please note that most of the links are private. If you don't have access to them, you'll get a 404 page! To gain access, you must be a Gruntwork customer. See gruntwork.io for more info and feel free to reach out to us at info@gruntwork.io if you have questions.

Terminology

Infrastructure Package: A reusable, tested, documented, configurable, best-practices definition of a single piece of Infrastructure (e.g., Docker cluster, VPC, Jenkins, Consul), written using a combination of Terraform, Go, and Bash. This code has been proven in production, providing the underlying infrastructure for Gruntwork's customers.
Module: Each Infrastructure Package consists one or more orthogonal modules that handle some specific aspect of that Infrastructure Package's functionality. Breaking the code up into multiple modules makes it easier to reuse and compose to handle many different use cases.
Reference Architecture: A best-practices way to combine all of Gruntwork's Infrastructure Packages into an end-to-end tech stack that contains just about all the infrastructure a company needs, including Docker clusters, databases, caches, load balancers, VPCs, CI servers, VPN servers, monitoring systems, log aggregation, alerting, secrets management, and so on. We build this all using infrastructure as code and immutable infrastructure principles, give you 100% of the code, and can get it deployed in minutes.

Infrastructure Packages

These are the Infrastructure Packages Gruntwork currently has available:

Network Topology: Create best-practices Virtual Private Clouds (VPCs) on AWS. The main modules are:
1. vpc-app: Launch a VPC meant to house applications and production code. This module creates the VPC, 3 "tiers" of subnets (public, private app, private persistence) across all Availability Zones, route tables, routing rules, Internet gateways, and NAT gateways.
2. vpc-mgmt: Launch a VPC meant to house internal tools (e.g. Jenkins, VPN server). This module creates the VPC, 2 "tiers" of subnets (public, private), route tables, routing rules, Internet gateways, and NAT gateways.
3. vpc-app-network-acls: Add a default set of Network ACLs to a VPC created using the vpc-app module that strictly control what inbound and outbound network traffic is allowed in each subnet of that VPC.
4. vpc-mgmt-network-acls: Add a default set of Network ACLs to a VPC created using the vpc-mgmt module that strictly control what inbound and outbound network traffic is allowed in each subnet of that VPC.
5. vpc-peering: Create peering connections between your VPCs to allow them to communicate with each other.
6. vpc-peering-external: Create peering connections between your VPCs and VPCs managed in other (external) AWS accounts.
Monitoring and Alerting: Configure monitoring, log aggregation, and alerting using CloudWatch, SNS, and S3. The main modules are:
1. alarms: A collection of more than 20 modules that set up CloudWatch Alarms for a variety of AWS services, such as CPU, memory, and disk space usage for EC2 Instances, Route 53 health checks for public endpoints, 4xx/5xx/connection errors for load balancers, and a way to send alarm notifications to a Slack channel.
2. logs: A collection of modules to set up log aggregation, including one to send logs from all of your EC2 instances to CloudWatch Logs, one to rotate and rate-limit logging so you don't run out of disk space, and one to store all load balancer logs in S3.
3. metrics: Modules that add custom metrics to CloudWatch, including critical metrics not visible to the EC2 hypervisor, such as memory usage and disk space usage.
Docker Cluster: Code to deploy and manage Docker containers on top of Amazon's EC2 Container Service (ECS). The main modules are:
1. ecs-cluster: Deploy an Auto Scaling Group (ASG) that ECS can use for running Docker containers. The size of the ASG can be scaled up or down in response to load.
2. ecs-service: Deploy a Docker container as a long-running ECS Service. Includes support for automated, zero-downtime deployment, auto-restart of crashed containers, and automatic integration with the Elastic Load Balancer (ELB).
3. ecs-service-with-alb: Deploy a Docker container as a long-running ECS Service. Includes support for automated, zero-downtime deployment, auto-restart of crashed containers, and automatic integration with the Application Load Balancer (ALB).
4. ecs-deploy: Deploy a Docker container as a short-running ECS Task, wait for it to exit, and exit with the same exit code as the ECS Task.
AMI Cluster: Deploy a best-practices Auto Scaling Group (ASG) that can run a cluster of servers, automatically restart failed nodes, and scale up and down in response to load. The main modules are:
1. asg-rolling-deploy: Create an ASG that can do a zero-downtime rolling deployment.
2. server-group: Run a fixed-size cluster of servers, backed by ASGs, that can automatically attach EBS Volumes and ENIs, while still supporting zero-downtime deployment.
Load Balancer: Run a highly-available and scalable load balancer in AWS. The main modules are:
1. alb: Deploy an Application Load Balancer (ALB) in AWS. It supports HTTP, HTTPS, HTTP/2, WebSockets, path-based routing, host-based routing, and health checks.
2. nlb: Deploy a Network Load Balancer (NLB) in AWS. It supports TCP, WebSockets, static IPs, high throughputs, and health checks.
Lambda: Deploy and manage AWS Lambda functions. The main modules are:
1. lambda: Deploy and manage AWS Lambda functions. Includes support for automatically uploading your code to AWS, configuring an IAM role for your Lambda function, and giving your Lambda function access to your VPCs.
2. scheduled-lambda-job: Configure your Lambda function to run on a scheduled basis, like a cron job.
API Gateway: Modules for deploying and managing Serverless Application Model applications with Lambda, API Gateway, and Terraform.
1. gruntsam: CLI tool that allows you to define your APIs using Swagger, run and test your code locally using SAM, and deploy your code to production using API Gateway and Lambda.
2. api-gateway-account-settings: set the global (regional) settings required to allow API Gateway to write to CloudWatch logs.
Security: Best practices for managing secrets, credentials, servers, and users. The main modules are:
1. auto-update: Configure your servers to automatically install critical security patches.
2. aws-auth: A script that makes it much easier to use the AWS CLI with MFA and/or multiple AWS accounts.
3. cloudtrail: Configure CloudTrail in an AWS account to audit all API calls.
4. kms-master-key: Create a master key in Amazon's Key Management Service and configure permissions for that key.
5. ssh-iam: Manage SSH access to your servers using IAM groups. Every developer in an IAM group you specify will be able to SSH to your servers using their own username and SSH key.
6. ssh-iam-selinux-policy: Install a SELinux Local Policy Module that is necessary to make ssh-iam work on systems with SELinux, such as CentOS.
7. iam-groups: Create a best-practices set of IAM groups for managing access to your AWS account.
8. iam-user-password-policy: Set the AWS Account Password Policy that will govern password requirements for IAM Users.
9. cross-account-iam-roles: Create IAM roles that allow IAM users to easily switch between AWS accounts.
10. fail2ban: Install fail2ban on your servers to automatically ban malicious users.
11. os-hardening: Build a hardened Linux-AMI that implements certian CIS benchmarks.
12. ntp: Install and configures NTP on a Linux server.
13. ip-lockdown: Install ip-lockdown on your servers to automatically lock down access to specific IPs, such as locking down the EC2 metadata endpoint so only the root user can access it.
GruntKMS: A command-line tool that makes it very easy to manage application secrets using Amazon's Key Management Service (KMS). GruntKMS is written in Go and compiles into a standalone binary for every major OS.
Continuous Delivery: Automate common Continuous Integration (CI) and Continuous Delivery (CD) tasks, such as installing dependencies, running tests, packaging code, and publishing releases. The main modules are:
1. aws-helpers: Automate common AWS tasks, such as publishing AMIs to multiple regions.
2. build-helpers: Automate the process of building versioned, immutable, deployable artifacts, including Docker images and AMIs built with Packer.
3. circleci-helpers: Configure the CircleCI environment, including installing Go and configuring GOPATH.
4. iam-policies: Configure common IAM policies for CI servers, including policies for automatically pushing Docker containers to ECR, deploying Docker images to ECS, and using S3 for Terraform remote state.
5. terraform-helpers: Automate common CI tasks that involve Terraform, such as automatically updating variables in a .tfvars file.
6. ec2-backup: Run a Lambda function to make scheduled backups of EC2 Instances.
7. install-jenkins: Install Jenkins on a Linux server.
8. jenkins-server: Deploy a Jenkins server with an ASG, EBS Volume, ALB, and Route 53 settings.
9. precommit-hooks: A collection of pre-commit hooks for Terraform, bash, Go, and more.
Relational Database: Deploy and manage relational databases such as MySQL and PostgreSQL using Amazon's Relational Database Service (RDS). The main modules are:
1. rds: Deploy a relational database on top of RDS. Includes support for MySQL, PostgreSQL, Oracle, and SQL Server, as well as automatic failover, read replicas, backups, patching, and encryption.
2. aurora: Deploy Amazon Aurora on top of RDS. This is a MySQL-compatible database that supports automatic failover, read replicas, backups, patching, and encryption.
3. lambda-create-snapshot: Create an AWS Lambda function that runs on a scheduled basis and takes snapshots of an RDS database for backup purposes. Includes an AWS alarm that goes off if backup fails.
4. lambda-share-snapshot: An AWS Lambda function that can automatically share an RDS snapshot with another AWS account. Useful for storing your RDS backups in a separate backup account.
5. lambda-copy-shared-snapshot: An AWS Lambda function that can make a local copy of an RDS snapshot shared from another AWS account. Useful for storing yoru RDS backups in a separate backup account.
6. lambda-cleanup-snapshots: An AWS Lambda function that runs on a scheduled basis to clean up old RDS database snapshots. Useful to ensure you aren't spending lots of money storing old snapshots you no longer need.
Distributed Cache: Deploy and manage Redis or Memcached on Amazon's ElastiCache service. The main modules are:
1. redis: Deploy a Redis cluster on top of ElastiCache. Includes support for automatic failover, backup, patches, and cluster scaling.
2. memcached: Deploy a Memcached cluster on top of ElastiCache.
Stateful Server: Deploy and manage a single server server on AWS. This is mainly useful for "stateful" servers that store data on their local hard disk, such as WordPress or Jenkins. The main modules are:
1. single-server: Run a server in AWS and configure its IAM role, security group, optional Elastic IP Address (EIP), and optional DNS A record in Route 53.
2. attach-eni: Attach an Elastic Network Interface (ENI) to a server during boot. This is useful when you need to maintain a pool of IP addresses that remain static even as the underlying servers are replaced.
3. persistent-ebs-volume: Attach and mount an EBS Volume in a server during boot. This is useful when you want to maintain data on a hard disk even as the underlying server is replaced.
4. route53-helpers: Attach a DNS A record in Route 53 to a server during boot. This is useful when you want a pool of domain names that remain static even as the underlying servers are replaced.
Static Assets: Store and serve your static content (i.e., CSS, JS, images) using S3 and CloudFront. The main modules are:
1. s3-static-website: Create an S3 bucket to host a static website. Includes support for custom routing rules and custom domain names.
2. s3-cloudfront: Deploy a CloudFront distribution as a CDN in front of an S3 bucket. Includes support for custom caching rules, custom domain names, and SSL.
MongoDB Cluster: Run a production-ready MongoDB cluster in AWS. The main modules are:
1. install-mongodb: Install MongoDB and all of its dependencies on a Linux server.
2. run-mongodb: Boot up a mongod, mongos, or mongo config server. Meant to be used in the User Data of the servers in your MongoDB cluster.
3. mongodb-cluster: Run a MongoDB cluster using an Auto Scaling Group (ASG) and configure its IAM role, security group, EBS Volume, ENI, and private domain name.
4. backup-mongodb: Back up the data in your MongoDB cluster to S3.
5. init-mongodb: Configure a MongoDB replica set.
OpenVPN Server: Deploy an OpenVPN server and control access to it using IAM. The main modules are:
1. install-openvpn: Install OpenVPN and its dependencies on a Linux server.
2. init-openvpn: Initialize an OpenVPN server, including its Public Key Infrastructure (PKI), Certificate Authority (CA) and configuration.
3. openvpn-admin: A command-line utility that allows users to request new certificates, administrators to revoke certificates and the OpenVPN server to process those requests. All access and permissions are controlled via IAM.
4. openvpn-server: Deploy an OpenVPN server and configure its IAM role, security group, Elastic IP Address (EIP), S3 bucket for storage, and SQS queues.
Messaging: Manage AWS Simple Queue Service (SQS) queues, AWS Simple Notification Service (SNS) topics, and AWS Kinesis streams. The main modules are:
1. sqs: Create an SQS queue. Includes support for FIFO, dead letter queues, and IP-limiting.
2. sns: Create an SNS topic as well as the publisher and subscriber policies for that topic.
3. kinesis: Create a Kinesis stream and configure its sharding settings.
Consul: Deploy and manage a Consul cluster on AWS. The main modules are:
1. install-consul: Install Consul and its dependencies on a Linux server.
2. install-dnsmasq: Install dnsmasq on a Linux server and configure it to work with Consul as a DNS server. This allows you to use domain names such as my-app.service.consul.
3. run-consul: Run Consul and automatically bootstrap the cluster.
4. consul-cluster: Deploy a Consul cluster on AWS using an Auto Scaling Group.
Nomad: Deploy and manage a Nomad cluster on AWS. The main modules are:
1. install-nomad: Install Nomad and its dependencies on a Linux server. 1 run-nomad: Run Nomad and automatically connect to a Consul cluster.
2. nomad-cluster: Deploy a Nomad cluster on AWS using an Auto Scaling Group.
Vault: Deploy and manage a Vault cluster on AWS. The main modules are:
1. install-vault: Install Vault and its dependencies on a Linux server.
2. run-vault: Run Vault and automatically connect to a Consul cluster.
3. private-tls-cert: Generate self-signed TLS certificates for use with Vault.
4. vault-cluster: Deploy a Vault cluster on AWS using an Auto Scaling Group.
ZooKeeper: Deploy and manage a ZooKeeper cluster, along with Exhibitor, on AWS. The main modules are:
1. install-zookeeper: Install ZooKeeper and its dependencies on a Linux server.
2. install-exhibitor: Install Exhibitor and its dependencies on a Linux server.
3. zookeeper-cluster: Deploy a ZooKeeper cluster using the server-group module from the AMI Cluster Infrastructure Package. Includes support for EBS Volumes for the transaction log as well as a set of ENIs to provide static IP addresses to ZooKeeper clients.
Kafka: Deploy and manage a cluster of Kafka brokers on AWS. The main modules are:
1. install-kafka: Install Kafka and its dependencies on a Linux server.
2. kafka-cluster: Deploy a Kafka cluster using the server-group module from the AMI Cluster Infrastructure Package. Includes support for EBS Volumes for the Kafka log.
package-terraform-utilities: Useful Terraform utilities. The main modules are:
1. intermediate-variable: A way to define intermediate variables in Terraform.
2. join-path: Join a list of given path parts (that is, file and folder names) into a single path with the appropriate path separator (backslash or forward slash) for the current operating system. 1 operating-system: Figure out what operating system is being used to run Terraform from inside your Terraform code.
terratest: The swiss army knife of testing Terraform modules. This is a library written in Go that we use to test all of the code above. It contains a collection of useful utilities: e.g., apply and destroy Terraform code, SSH to servers and run commands, test HTTP endpoints, fetch data from AWS, build AMIs using Packer, run Docker builds, and so on.

Reference Architecture

The Gruntwork Reference Architecture is a best-practices way to combine all of Gruntwork's Infrastructure Packages into an end-to-end tech stack that contains just about all the infrastructure a company needs, including Docker clusters, databases, caches, load balancers, VPCs, CI servers, VPN servers, monitoring systems, log aggregation, alerting, secrets management, and so on. We build this all using infrastructure as code and immutable infrastructure principles, give you 100% of the code, and can get it deployed in minutes.

You can view the Reference Architecture for a fictional company, Acme, in one of two flavors:

Single-account reference architecture
Multi-account reference architecture

Single-account reference architecture

In a single account setup, all environments (e.g., stage, prod, etc) are deployed in a single AWS account, albeit, in separate VPCs. This gives you ease-of-use and convenience, but not as much isolation/security.

infrastratructure-modules: The reusable modules that define the infrastructure for the entire company.
infrastratructure-live: Use the modules in infrastructure-modules to deploy all of the live environments for the company.
sample-app-frontend: A sample app that demonstrates best practices for an individual Docker-based frontend app or microservice that talks to backend apps (showing how to do service discovery) and returns HTML. This app is written in NodeJS but these practices are broadly applicable (and in the case of Docker, reusable!) across any technology platform.
sample-app-backend: A sample app that demonstrates best practices for an individual Docker-based backend app or microservice that talks to a database. This app is written in NodeJS but these practices are broadly applicable (and in the case of Docker, reusable!) across any technology platform.

Multi-account reference architecture

In a multi-account setup, each environment (e.g., stage, prod, etc) is deployed into a separate AWS account, and in its own VPC within that account. All IAM users are defined in yet another AWS account called security, and you can assume IAM roles to switch between accounts. This gives you much more fine-grained control over security, and complete isolation between accounts, but there it's less convenient to use, as you have to keep switching between accounts.

infrastratructure-modules: The reusable modules that define the infrastructure for the entire company.
infrastratructure-live: Use the modules in infrastructure-modules to deploy all of the live environments for the company.
sample-app-frontend: A sample app that demonstrates best practices for an individual Docker-based frontend app or microservice that talks to backend apps (showing how to do service discovery) and returns HTML. This app is written in NodeJS but these practices are broadly applicable (and in the case of Docker, reusable!) across any technology platform.
sample-app-backend: A sample app that demonstrates best practices for an individual Docker-based backend app or microservice that talks to a database. This app is written in NodeJS but these practices are broadly applicable (and in the case of Docker, reusable!) across any technology platform.

Gruntwork Open Source Tools

terragrunt: A thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules.
fetch: A tool that makes it easy to download files, folders, and release assets from a specific git commit, branch, or tag of public and private GitHub repos.
gruntwork-installer: A script to make it easy to install Gruntwork Modules.

Gruntwork Training Reference Materials

infrastructure-as-code-training: A sample repo we share with customers when we do training on Terraform, Docker, Packer, AWS, etc.

JuulLabs / toc