This project has the objective of teaching you how to build a Custom CDK Pipeline, but mainly teach you how to build your own Custom CDK Constructs. You'll additionally learn how to implement unit tests for your Construct, and even insert those tests in your deployment pipeline.
Before you can dive into this project, you should be familiar with a few concepts and AWS Services:
- DevOps
- Infrastructure as Code
- CI/CD
- Tests (more specifically Unit Tests)
- Python
- AWS Cloud Development Kit (CDK)
- AWS CloudFormation
- AWS CodePipeline
- AWS CodeBuild
- AWS CodeCommit
- Amazon API Gateway
- AWS Lambda
When you deploy this application, you'll have a few new resources created. Here are the most important ones:
- A
masterpipeline in CodePipeline, alongside 2 CodeBuild projects - A
developpipeline in CodePipeline, alongside 2 CodeBuild projects - An API Gateway
- A 'Hello World' Lambda Function
The API Gateway and Lambda Function are part of the Application Construct. You can change everything inside here to exactly represent the resources needed for your application.
The master and develop pipelines are part of the Pipeline Construct. If you need specific deployment steps,
additional stages, etc., you should implement your changes here.
Finally, we orchestrate the creation of our application and our pipeline in the Custom Pipeline Stack. Here's where we reference our Git repository (in this case, CodeCommit). You should change this information so your pipelines can point to your repository. In case you want to use GitHub as your source repository, you can check this documentation or follow this example.
In order to have totally separate deployments, we're going to have 2 separate stacks, one for the production environment
(master branch), and another one for the development environment (develop branch). This way, if you perform changes
to the develop branch, only the development pipeline is going to be triggered, and therefore, only the development
environment is going to be affected.
It's nice to say that this separation could also have been implemented in a few different ways, like with Nested
Stacks,
with another centralized Construct, etc., the final solution would be the same.
Quoting the AWS CDK service page:
The AWS Cloud Development Kit (AWS CDK) is an open source software development framework to define your cloud application resources using familiar programming languages.
What happens "under the hood" when you deploy an AWS CDK Stack is a conversion of everything that you wrote in your chosen language to a CloudFormation template, and then AWS CDK automatically runs that template in CloudFormation for you.
To see more on how AWS CDK has introduced us the concept of 'Infrastructure is Code' instead of the usual 'Infrastructure as Code', check the AWS CDK developer guide, and also check its API reference.
Additionally, it's highly recommended that you check out the AWS CDK Workshop. There, you're going to learn how to use AWS CDK to build your own stacks, and even begin to learn the concept of Constructs.
If you checked out the AWS CDK Workshop mentioned in the previous topic, you've probably already done this. But in case you didn't, this is how you install AWS CDK:
-
First, check out the pre-requisites page in the Developer Guide.
-
Then, run the following command to install the AWS CDK Toolkit:
$ npm install -g aws-cdk -
Check if the AWS CDK Toolkit was installed successfully:
$ cdk --version 1.21.1 (build 842cc5f) -
Before you run this project, you need to bootstrap AWS CDK to create the needed resources for the toolkit’s operation in you AWS Account. So run the command below:
$ cdk bootstrap -
Go to the AWS CodeCommit dashboard in your AWS account and create a repository (if you want, use the name
custom-cdk-pipeline-constructto skip step 8). -
Clone the AWS CodeCommit repository you've just created to your local machine.
-
Download this GitHub repository's code and unzip all its content inside your AWS CodeCommit repo's local folder.
-
If you didn't use
custom-cdk-pipeline-constructfor your repo's name, open thecdk_pipeline_artifact/custom_pipeline_stack.pyfile and change the repository name according to the one you created in your account:app_repository = codecommit.Repository.from_repository_name(self, 'CodeCommitRepo', '<YOUR_REPO_NAME_HERE>') -
Commit and push everything to your AWS CodeCommit repository.
-
Finally, deploy the project:
$ cdk deploy --all
If we only had 1 stack to be created, we could simply run cdk deploy. However, in this case we have 2 stacks, so we
need to specify --all to deploy all existing stacks. We could also deploy only 1 of our stacks if desired, running
cdk deploy <stack-name>.
Quoting the AWS CDK Developer Guide:
Constructs are the basic building blocks of AWS CDK apps. A construct represents a "cloud component" and encapsulates everything AWS CloudFormation needs to create the component.
A construct can represent a single resource, such as an Amazon Simple Storage Service (Amazon S3) bucket, or it can represent a higher-level component consisting of multiple AWS resources. Examples of such components include a worker queue with its associated compute capacity, a cron job with monitoring resources and a dashboard, or even an entire app spanning multiple AWS accounts and regions.
The AWS CDK includes the AWS Construct Library, which contains constructs representing AWS resources.
This library includes constructs that represent all the resources available on AWS. For example, the s3.Bucket class represents an Amazon S3 bucket, and the dynamodb.Table class represents an Amazon DynamoDB table.
To keep diving inside this project, you should also be familiar with the different Construct levels. So let's continue reading the API Reference:
There are three different levels of constructs in this library, beginning with low-level constructs, which we call CFN Resources (or L1, short for "level 1") or Cfn (short for CloudFormation) resources. These constructs directly represent all resources available in AWS CloudFormation. CFN Resources are periodically generated from the AWS CloudFormation Resource Specification. They are named CfnXyz, where Xyz is name of the resource. For example, CfnBucket represents the AWS::S3::Bucket AWS CloudFormation resource. When you use Cfn resources, you must explicitly configure all resource properties, which requires a complete understanding of the details of the underlying AWS CloudFormation resource model.
The next level of constructs, L2, also represent AWS resources, but with a higher-level, intent-based API. They provide similar functionality, but provide the defaults, boilerplate, and glue logic you'd be writing yourself with a CFN Resource construct. AWS constructs offer convenient defaults and reduce the need to know all the details about the AWS resources they represent, while providing convenience methods that make it simpler to work with the resource. For example, the s3.Bucket class represents an Amazon S3 bucket with additional properties and methods, such as bucket.addLifeCycleRule(), which adds a lifecycle rule to the bucket.
Finally, the AWS Construct Library includes even higher-level constructs, L3, which we call patterns. These constructs are designed to help you complete common tasks in AWS, often involving multiple kinds of resources. For example, the aws-ecs-patterns.ApplicationLoadBalancedFargateService construct represents an architecture that includes an AWS Fargate container cluster employing an Application Load Balancer (ALB). The aws-apigateway.LambdaRestApi construct represents an Amazon API Gateway API that's backed by an AWS Lambda function.
In this project, you'll see that 2 L3 Constructs were built, the ApplicationConstruct and the CustomPipeline. Within
those, we've used 2 L3 constructs provided by AWS:
aws_solutions_constructs.aws_apigateway_lambda.ApiGatewayToLambda: This Construct builds an Amazon API Gateway, with an endpoint directed to an AWS Lambda Function, while enabling the CloudWatch logs and X-Ray tracing for both API Gateway and Lambda. Check out the library's documentation.aws_cdk.pipelines.CdkPipeline: This Construct creates a pipeline in AWS CodePipeline to deploy AWS CDK applications. One interesting feature of this library is that is has aSelfMutatestep in the end of the pipeline, that deploys your AWS CDK application and mutates the pipeline itself in case there were changes made to the it. Check out its documentation.
Creating a custom Construct it a lot easier than you might think. Simply create a new class, and use the
aws_cdk.core.Construct as its parent. That's it! See the ApplicationConstruct example:
class ApplicationConstruct(core.Construct):
# ...
After you do this, you can start implementing anything you want inside your Construct. In our ApplicationConstruct
example, we implemented 2 possibilities, the creation of the API Gateway with IAM Authorization or with no
authorization at all. This is not a trivial thing to do inside the aws_solutions_constructs.aws_apigateway_lambda.ApiGatewayToLambda
library, so we simplified this by creating our Construct and just receiving an use_iam_authorization parameter that,
under the hood, will do the configurations needed for each API Gateway authorization type.
Right here we've created an L3 Construct, that uses another L3 Construct internally.
Now, imagine you have a bigger and more complex application, like an ELB in front of an ECS cluster that connects to a DynamoDB table and saves files to an S3 bucket, and all of that behind a CloudFront distribution with a Route 53 domain. You could implement all that infrastructure inside your own Construct, and even if you needed several different environments, you could simply instantiate your Construct several times, and all your infrastructure would be ready for all your environments.
That's exactly what we do with our CustomPipeline Construct! We created a tool that manages several resources, but can
be quickly and easily replicated. So let's check how we've done that.
Here, we've followed the same pattern we did in our ApplicationConstruct:
class CustomPipeline(core.Construct):
# ...
In order to build a pipeline in AWS CodePipeline, we must create several stages:
- Reference our source Git repository in CodeCommit
- Run our unit tests and build our CDK Application
- Deploy the application and redeploy the pipeline itself in case anything changed
To achieve that, we've created several resources. So let's begin with our first stage.
Here we created a source_artifact (aws_cdk.aws_codepipeline.Artifact) to store all our repositories files during the
deployment. After that, we created a source_action (aws_cdk.aws_codepipeline_actions.CodeCommitSourceAction) using
the repository and branch provided to us, and referencing the source_artifact we've just created.
To satisfy the second stage, we've built a cloud_assembly_artifact (aws_cdk.aws_codepipeline.Artifact) to store the
CloudFormation file that is going to be generated here and deployed in our third stage. Then, we created the
synth_action (aws_cdk.pipelines.SimpleSynthAction), which creates a CodeBuild project. The project has an
install_command step, where we can install any libraries needed for the test_commands and synth_command steps.
Next, the project runs the test_commands step, which will execute any tests implemented (we'll explain more about that
in the [Testing your custom Construct](#Testing your custom Construct) section of this document). And finally, in the
synth_command step it builds our application, which is nothing more than running npx cdk synth to create the
CloudFormation template that will represent the infrastructure of our application.
Lastly, to satisfy the final stage, we created the cdk_pipeline (aws_cdk.pipelines.CdkPipeline), that will actually
create the pipeline in CodePipeline, and reference the 2 stages created previously. As we're creating our pipeline using
the CdkPipeline, in the end of it we're going to additionally have the SelfMutate stage, mentioned in the end of the
[What is a Construct?](#What is a Construct?) section of this document. The SelfMutate stage deploys the AWS
CloudFormation template produced in the synth_action, deploying our application and consequently redeploying the
pipeline itself. To do so, this library uses prerelease features of the AWS CDK framework, which can be enabled by
adding the following code to the cdk.json of your project:
```
{
// ...
"context": {
"@aws-cdk/core:newStyleStackSynthesis": true
}
}
```
If you don't do this procedure, the AWS CodeBuild project will not have the correct permissions in its AWS IAM Role, and
will always fail in the SelfMutate step. You can see more information regarding the CdkPipeline library in
this documentation.
Finally, in the creation of the CdkPipeline, we also reference the created cloud_assembly_artifact, so during the
SelfMutate stage it can get the CloudFormation template generated in the Build stage and deploy it.
With all that said, this might seem like a lot, and that's exactly why we created this Construct. So in the future,
instead of copying and pasting almost 30 lines of code, you can simply instantiate a CustomPipeline object, and you'll
have a totally working pipeline for the desired repository and branch.
Tests are an important part of software development and CI/CD, so let's focus right now on how to implement unit tests
for our CDK Pipeline CustomPipeline Construct.
For our pipeline to be created successfully, all of the components mentioned in the [Building a Custom CDK Pipeline with Constructs](#Building a Custom CDK Pipeline with Constructs) section of this document must be created. Therefore, those are exactly the points we need to test: if every component was created the way they should be.
In unit tests we only test our code, we shouldn't test external libraries, so we need to mock every component from the
aws_cdk library and any other external library we use. That's why if you check the
test_custom_cdk_pipeline.TestPipelineConstruct.test_should_break_if_source_action_not_created_properly for example,
you'll see that the artifacts, repository and even the source action are mocked.
Now let's begin understanding the unit tests implementation for our Construct. There's 3 necessary things for us to test:
- The
source_action, that'll fetch our code from the CodeCommit repository - The
simple_synth_action, that'll build our CDK application and generate the CloudFormation template - The
cdk_pipeline, that'll create our pipeline in CodePipeline and add theSelfMutatestep in the end of it to deploy our app and update itself
We need to make sure that all of these components are created only once and with the correct parameters.
So let's start with the source_action. As previously mentioned, we mocked the artifacts, repository and even the
source action.
An interesting and necessary approach we took here is the mock.MagicMock.side_effect. As we created 2
artifacts (aws_cdk.aws_codepipeline.Artifact), we need to make sure that each one of them is passed correctly in the
source action creation. So when we assign a list composed by the mocked artifacts, in their respective order, to the
side_effect property of the mocked Artifact (aws_cdk.aws_codepipeline.Artifact), we're inferring that whenever an
Artifact is created, its value will be retrieved from this list. Check out the unittest.mock documentation
for more details on the side_effect feature.
Now there's an important thing we have to talk about. As we've said before, in unit tests we only test our code, we
shouldn't test external libraries. However, when testing Constructs we need to create an actual Stack
(aws_cdk.core.Stack) in order for it to work. If we try to mock the Stack object as well, the Construct creation
will fail due to how the CDK code architecture works, and none of our tests will run. For more information regarding
testing Constructs, check out this section of the Developer Guide.
After creating the Stack, we need to instantiate our actual CustomPipeline Construct to check if the expected values
in our tests comply with what's implemented. Here's where we need the Stack we created, to pass as an argument when
instantiating our Construct.
With all these things in mind, let's finally test the source_action_mock. Here we'll have to test if it was called
only once and if we're passing the expected parameters to it. To satisfy this, we'll use the
mock.MagicMock.assert_called_once_with method, giving the parameters we expect to have been passed during the
execution:
- action_name: can be any desired name
- repository: the mocked
repositorywe created - branch: the branch name for our test, which can also be any desired name
- output: the mocked
source_artifactwe created
To read more about the assert_called_once_with, check out the unittest.mock documentation.
To test the simple_synth_action we're going to implement some similar snippets we did in the [Source Action](#Source Action)
section, like mocking the artifacts and repository, using the side_effect property for the artifacts, creating the
Stack and invoking the actual code. The main difference is that we'll be testing the simple_synth_action_mock.
Again, we'll use the assert_called_once_with, but here we'll use the needed parameters for the
simple_synth_action_mock:
- action_name: can be any desired name
- source_artifact: the mocked
source_artifactwe created - cloud_assembly_artifact: the
cloud_assembly_artifactwe created - test_commands: the commands that will run our tests. Check the [Running our unit tests](#Running our unit tests) section of this document for more information.
- install_command: the command to install all necessary programs and libraries for our build process to work
- synth_command: the command that will synthesise our CloudFormation template
Finally, to test the cdk_pipeline_mock we'll also need similar snippets we implemented before, as mentioned in the
beginning of the [Simple Synth Action](#Simple Synth Action) section. However, here we'll need to use another feature
from the unittest lib.
In the cdk_pipeline_mock we'll need to pass as a parameter the object returned by the mocked source_action. To
achieve this, we'll use the return_value property of our source_action_mock and assign the
source_action_value_mock to it. This way, when our actual code tries to get the value of source_action_mock, it will
receive the assigned mocked value.
We'll also need to do the same thing with the simple_synth_action_mock.
Finally, let's test the cdk_pipeline_mock. Here we'll use the assert_called_once_with again, but with the needed
parameters for the cdk_pipeline_mock:
- scope: the
custom_pipelinewe created - construct_id: the test construct id, which can be any desired name
- pipeline_name: the test pipeline name, which can be any desired name
- cloud_assembly_artifact: the
cloud_assembly_artifact_mockwe created - source_action: the
source_action_value_mockwe created - synth_action: the
simple_synth_action_value_mockwe created
When using the unittest library, you can simply run your tests with the following command:
$ python -m unittest
The library will automatically locate the implemented tests inside the project and run them.
When running your tests, if you get a message similar with the one below, all your tests were successful:
.....
----------------------------------------------------------------------
Ran 5 tests in 0.018s
OK
However, if the message is similar to the one below, one or more of your tests have failed:
....F
======================================================================
FAIL: test_should_break_if_synth_action_not_created_properly (tests.unit.test_custom_cdk_pipeline.TestPipelineConstruct)
----------------------------------------------------------------------
Traceback (most recent call last):
...
AssertionError: expected call not found.
Expected: SimpleSynthAction(action_name='Cdk_Build', source_artifact=<MagicMock id='4368728976'>, cloud_assembly_artifact=<MagicMock id='4430777872'>, test_commands=['python -m unittest'], install_command='npm install -g aws-cdk && python -m pip install -r requirements.txt', synth_command='npx cdk synth')
Actual: SimpleSynthAction(action_name='Cdk_Build', source_artifact=<MagicMock id='4368728976'>, cloud_assembly_artifact=<MagicMock id='4430777872'>, test_commands=['python -m unittest'], install_command='npm install -g aws-cdk && python -m pip install -r requirements.txt', synth_command='npx cdk diff')
----------------------------------------------------------------------
Ran 5 tests in 0.022s
FAILED (failures=1)
In the message above, you can see the reason of why this specific test failed. You will receive similar messages for each failed test. If the tests fail during the build stage of our pipeline, the whole pipeline deployment will automatically fail as well.
When building the synth_action for our pipeline, we can specify the test_commands, just like we mentioned in the
[Building a Custom CDK Pipeline with Constructs](#Building a Custom CDK Pipeline with Constructs) section of this
document. That's where you should insert the python -m unittest command to test your application during the build
stage.
We've finally reached the end of our project. Here, we've shown you how to build a custom Construct from scratch, how to build your own CDK Pipeline Construct, how to implement unit tests for it and how to insert those tests in your pipeline. We thoroughly went step by step into each of these phases, showing why each piece of code was implemented, and how you could replicate it to create your own custom Construct.
I hope this artifact was helpful and that you've learned something new!
See CONTRIBUTING for more information.
This library is licensed under the MIT-0 License. See the LICENSE file.