jdfalko/Trapheus

AWS RDS Snapshot Restoration
Pre-Requisites
Parameters
Instructions
Execution
How it Works
- AWS Step Functions State Machine
Contributing to Trapheus

With Trapheus, it becomes easy to get and restore database instances from snapshots into any dev, staging or production environments. Trapheus supports individual RDS Snapshot as well as cluster snapshot restore operations. Trapheus is modelled as a state machine, with the help of AWS step functions, that executes different steps in order to restore an existing database from a snapshot. Trapheus can be scheduled using a cloudwatch alarm rule or can be run using any of the supported AWS step function triggers.

Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the AWS pricing page for details.

├── CONTRIBUTING.md                               <-- How to contribute to Trapheus
├── LICENSE.md                                    <-- The MIT license.
├── README.md                                     <-- The Readme file.
├── events
│   └── event.json                                <-- JSON event file to be used for local SAM testing
├── screenshots                                   <-- Folder for screenshots of teh state machine.
│   ├── Trapheus-logo.png
│   ├── cluster_restore.png
│   ├── cluster_snapshot_branch.png
│   ├── failure_handling.png
│   ├── instance_restore.png
│   ├── instance_snapshot_branch.png
│   ├── isCluster_branching.png
│   └── restore_state_machine.png
├── src
│   ├── checkstatus
│   │   ├── DBClusterStatusWaiter.py              <-- Python Waiter(https://boto3.amazonaws.com/v1/documentation/api/latest/guide/clients.html#waiters) for checking the status of the cluster
│   │   ├── get_dbcluster_status_function.py      <-- Python Lambda code for polling the status of a clusterised database
│   │   ├── get_dbstatus_function.py              <-- Python Lambda code for polling the status of a non clusterised RDS instance
│   │   └── waiter_acceptor_config.py             <-- Config module for the waiters
│   ├── common                                    <-- Common modules across the state machine deployed as a AWS Lambda layer.
│   │   ├── common.zip
│   │   └── python
│   │       ├── constants.py                      <-- Common constants used across the state machine.
│   │       ├── custom_exceptions.py              <-- Custom exceptions defined for the entire state machine.
│   │       └── utility.py                        <-- Utility module.
│   ├── delete
│   │   ├── cluster_delete_function.py           <-- Python Lambda code for deleting a clusterised database.
│   │   └── delete_function.py                   <-- Python Lambda code for deleting a non clusterised RDS instance.
│   ├── emailalert
│   │   └── email_function.py                    <-- Python Lambda code for sending out failure emails.
│   ├── rename
│   │   ├── cluster_rename_function.py           <-- Python Lambda code for renaming a clusterised database.
│   │   └── rename_function.py                   <-- Python Lambda code for renaming a non-clusterised RDS instance.
│   ├── restore
│   │   ├── cluster_restore_function.py          <-- Python Lambda code for retoring a clusterised database.
│   │   └── restore_function.py                  <-- Python Lambda code for restoring a non-clusterised RDS instance
│   ├── slackNotification
│   │   └── slack_notification.py                <-- Python Lambda code for sending out a failure alert to configured webhook(s) on Slack.
│   └── snapshot
│       ├── cluster_snapshot_function.py         <-- Python Lambda code for creating a snapshot of a clusterised database.
│       └── snapshot_function.py                 <-- Python Lambda code for creating a snapshot of a non-clusterised RDS instance.
├── template.yaml                                <-- SAM template definition for the entire state machine.
└── tests                                        <-- Test folder.
    └── unit
        ├── mock_constants.py
        ├── mock_custom_exceptions.py
        ├── mock_import.py
        ├── mock_utility.py
        ├── test_cluster_delete_function.py
        ├── test_cluster_rename_function.py
        ├── test_cluster_restore_function.py
        ├── test_cluster_snapshot_function.py
        ├── test_delete_function.py
        ├── test_email_function.py
        ├── test_get_dbcluster_status_function.py
        ├── test_get_dbstatus_function.py
        ├── test_rename_function.py
        ├── test_restore_function.py
        ├── test_slack_notification.py
        └── test_snapshot_function.py

Pre-Requisites

The app requires the following AWS resources to exist before installation:

python3.7 installed on local machine following this.
Have sam installed in the local machine. Can be installed following the Installing the AWS SAM CLI documentation.
Configure AWS SES
- Configure the SES sender and receiver email (SES Console->Email Addresses).
  - An SES email alert is configured to notify the user about any failures in the state machine. The sender email parameter is needed to configure the email ID through which the alert is sent out. The receiver email parameter is needed to set the email ID to which the alert is sent.
Create the S3 bucket where the system is going to store the cloud formation templates:
- Proposed Name: trapheus-cfn-s3-[account-id]-[region]. It is recommended that the name contains your:
  - account-id, as the bucket names need to be global (prevents someone else having the same name)
  - region, to easily keep track when you have trapheus-s3 buckets in multiple regions
A VPC (region specific). The same VPC/region should be used for both the RDS instance(s), to be used in Trapheus, and Trapheus' lambdas.
- Region selection consideration. Regions that support:
  - Email receiving . Check Parameters -> 'RecipientEmail' for more.
- Example minimal VPC setup:
  - VPC console:
    - name: Trapheus-VPC-[region] (specify the [region] where you VPC is created - to easily keep track when you have Trapheus-VPCs in multiple regions)
    - IPv4 CIDR block: 10.0.0.0/16
  - VPC console->Subnets page and create two private subnets:
    - Subnet1:
      - VPC: Trapheus-VPC-[region]
      - Availability Zone: choose one
      - IPv4 CIDR block: 10.0.0.0/19
    - Subnet2:
      - VPC: Trapheus-VPC-[region]
      - Availability Zone: choose a different one than the Subnet1 AZ.
      - IPv4 CIDR block: 10.0.32.0/19
  - You have created a VPC with only two private subnets. If you are creating non-private subnets, check the ratio between private, public subnets, private subnet with dedicated custom network ACL and spare capacity.
One or more instances of an RDS database that you wish to restore.
- Example minimal free RDS setup:
  - Engine options: MySQL
  - Templates: Free tier
  - Settings: enter password
  - Connectivity: VPC: Trapheus-VPC-[region]

Parameters

The following are the parameters for creating the cloudformation template:

--s3-bucket : [Optional] The name of the CloudFormation template S3 bucket from the Pre-Requisites.
vpcID : [Required] The id of the VPC from the Pre-Requisites. The lambdas from the Trapheus state machine will be created in this VPC.
Subnets : [Required] A comma separated list of private subnet ids (region specific) from the Pre-Requisites VPC.
SenderEmail : [Required] The SES sending email configured in the Pre-Requisites
RecipientEmail : [Required] Comma separated list of recipient email addresses configured in Pre-Requisites.
UseVPCAndSubnets : [Optional] Whether to use the vpc and subnets to create a security group and link the security group and vpc to the lambdas. When UseVPCAndSubnets left out (default) or set to 'true', lambdas are connected to a VPC in your account, and by default the function can't access the RDS (or other services) if VPC doesn't provide access (either by routing outbound traffic to a NAT gateway in a public subnet, or having a VPC endpoint, both of which incur cost or require more setup). If set to 'false', the lambdas will run in a default Lambda owned VPC that has access to RDS (and other AWS services).
SlackWebhookUrls : [Optional] Comma separated list of Slack webhooks for failure alerts.

Instructions

Setup

To setup the Trapheus in your AWS account, follow the steps below:

Clone the Trapheus Git repository
From the Trapheus repo execute:
sam package --template-file template.yaml --output-template-file deploy.yaml --s3-bucket <s3 bucket name from corresponding AWS account and region>
To deploy the stack using the above mentioned parameters execute:
sam deploy --template-file deploy.yaml --stack-name <user-defined-stack-name> --region <aws region> --capabilities CAPABILITY_NAMED_IAM --parameter-overrides vpcId=<vpcID> Subnets=<Subnets> SenderEmail=<SenderEmail> RecipientEmail=<RecipientEmail>
3.1. In order to have the minimal Lambda-RDS access configuration and less costs (UseVPCAndSubnets description above for more), add the UseVPCAndSubnets=false at the end of the sam deploy command or:
sam deploy --template-file deploy.yaml --stack-name <user-defined-stack-name> --region <aws region> --capabilities CAPABILITY_NAMED_IAM --parameter-overrides vpcId=<vpcID> Subnets=<Subnets> SenderEmail=<SenderEmail> RecipientEmail=<RecipientEmail> UseVPCAndSubnets=false
Typically, linking your own VPC to the lambdas and not setting addition NAT gateway or VPC endpoint configuration will result in a "Connect timeout on endpoint URL: "https://rds.[region].amazonaws.com/"" or "Task timed out after 3.00 seconds" issue.
3.2. [Optional] To add Slack notification for failure alerts, add the parameter SlackWebhookUrls to the end of the deploy command like so:
sam deploy --template-file deploy.yaml --stack-name <user-defined-stack-name> --region <aws region> --capabilities CAPABILITY_NAMED_IAM --parameter-overrides vpcId=<vpcID> Subnets=<Subnets> SenderEmail=<SenderEmail> RecipientEmail=<RecipientEmail> --SlackWebhookUrls=<comma-separated-slack-webhook-urls>
More information about setting up Slack webhooks can be found here

TO BE NOTED: The CFT creates the following resources:

DBRestoreStateMachine Step function state machine
Multiple lambdas to execute various steps in the state machine
LambdaExecutionRole: used across all lambdas to perform multiple tasks across RDS
StatesExecutionRole: IAM role with permissions for executing the state machine and invoking lambdas
S3 bucket: rds-snapshots-<your_account_id> where snapshots will be exported to
KMS key: is required to start export task of snapshot to s3

Execution

To execute the step function, follow the steps below:

Navigate to the State machine definition from the Resources tab in the cloudformation stack.
Click on Start Execution.
Under Input, provide the following json as parameter:

{
    "identifier": "<identifier name>",
    "task": "<taskname>",
    "isCluster": true or false
}

a. identifier: (Required - String) The RDS instance or cluster identifier that has to be restored. Any type of RDS instance or Amazon aurora clusters are supported in this.

b. task: (Required - String) Valid options are create_snapshot or db_restore.

c. isCluster: (Required - Boolean) Set to true if the identifier provided is of a cluster else set to false

The state machine can do one of the following tasks:

if task is set to create_snapshot, the state machine creates/updates a snapshot for the given RDS instance or cluster using the snapshot identifier: identifier-snapshot and then executes the pipeline
if task is set to db_restore, the state machine does a restore on the given RDS instance, without updating a snapshot, assuming there is an existing snapshot with an identifier: identifier-snapshot

Cost considerations

After done with development or using the tool:

if you don't need the RDS instance when not coding or using the tool (for instance, if it is a test RDS), consider stopping or deleting the database. You can always recreate it when you need it.
if you don't need the past Cloud Formation templates, it is recommended you empty the CFN S3 bucket.

Tear down

To tear down your application and remove all resources associated with the Trapheus DB Restore state machine, follow these steps:

Log into the Amazon CloudFormation Console and find the stack you created.
Delete the stack. Note that stack deletion will fail if rds-snapshots-<YOUR_ACCOUNT_NO> s3 bucket is not empty, so first delete the snapshots' exports in the bucket.
Delete the AWS resources from the Pre-Requisites. Removal of SES, the CFN S3 bucket (empty it if not deleting) and VPC is optional as you won't see charges, but can re-use them later for a quick start.

How it Works

Complete Pipeline

Modelled as a state machine, different steps in the flow such as snapshot creation/updation, instance rename, restore and deletion, completion/failure status of each operation, failure email alert, etc. are executed using individual lambdas for db instances and db clusters respectively. To track completion/failure of each operation, RDS waiters are used with delays and maximum retry attempts configured based on the lambda timeout. For scenarios of DB cluster availability and deletion, custom waiters have been defined. Lambda layers are used across all lambdas for common utility methods and custom exception handling.

Based on the input provided to the DBRestoreStateMachine step function, the following steps/branches are executed:

Using the isCluster value, a branching takes place in the state machine to execute the pipeline for a db cluster or for a db instance.
If task is set to create_snapshot, the snapshot creation/updation process takes place for a cluster or instance respectively. Creates a snapshot using the unique identifier: identifier-snapshot, if it does not exist. If a snapshot already exists with the aforementioned identifier, it is deleted and a new snapshot is created.
If task is set to db_restore, the db restoration process starts, without a snapshot creation/updation
As part of the db restoration process, the first step is a Rename of the provided db instance or db cluster and its corresponding instances to a temporary name. Wait for successful completion of the rename step to be able to use the provided unique identifier in the restoration step.
Once the rename step is complete, next step is to Restore the db-instance or db-cluster using the identifier parameter and the snapshot id as identifier-snapshot
Once the restore is complete and the db-instance or db-cluster is available, the final step is to Delete the initially renamed instance or cluster (along with its instances) which was retained for failure handling purposes. Executed using lambdas created for deletion purposes, once the deletion is successful, the pipeline is complete.
At any step, the retries with backoff and failure alerts are handled in every step of the state machine. If there is an occurrence of a failure, an SES email alert is sent as configured during the setup. Optionally, if SlackWebhookUrls was provided in the setup, failure notifications will also be sent to the appropriate channels.
If the restore step fails, as part of failure handling, the Step-4 of instance/cluster rename is reverted to ensure that the original db-instance or db cluster is available for use.

Contributing to Trapheus

Prepare your environment. Install tools as needed.

Git Bash used to run Git from Command Line.
Github Desktop Git desktop tool for managing pull requests, branches and repos.
Visual Studio Code Full visual editor. Extensions for GitHub can be added.
Or an editor of your choice.

Fork Trapheus repo
Create a working branch.
```
git branch trapheus-change1
```
Confirm working branch for changes.
```
 git checkout trapheus-change1
```
You can combine both commands by typing git checkout -b trapheus-change1.
Make changes locally using an editor and add unit tests as needed.

Run the test suite in the repo to ensure existing flows are not breaking.

   cd Trapheus
   python -m pytest tests/ -v #to execute the complete test suite
   python -m pytest tests/unit/test_get_dbstatus_function.py -v #to execute any individual test

Stage edited files.
```
   git add contentfile.md 
```
Or use git add . for multiple files.
Commit changes from staging.
```
   git commit -m "trapheus-change1"
```

Push new changes to GitHub

   git push --set-upstream origin trapheus-change1

Verify status of branch
```
git status
```
Review Output to confirm commit status.

Git push

    git push --set-upstream origin trapheus-change1

The Output will provide a link to create your Pull Request.

jdfalko / Trapheus