aws-ci-codepipeline

The AWS code pipeline for CI (i.e. build). It creates a codebuild project and S3 artifact bucket in a shared service account. The output artifact can be used to trigger the code pipeline for CD (i.e. deployment) in other AWS account(s). For multi-region build, modify the module to replicate the output artifact to an S3 bucket in the same region as the target deployment code pipeline. The module currently supports multi-region build for lambda, ECS and ECR in the N.Virginia and Ireland regions.

v1.0 Notes

The account that owns the github token must have admin access on the repo in order to generate a github webhook.
If use_docker_credentials is set to true, the environment variables DOCKERHUB_USER and DOCKERHUB_PASS are exposed via codebuild.

You can add these 2 lines to the beginning of your build phase commands in buildspec.yml to login to Dockerhub.
```
build:
    commands:
    - echo "Logging into Dockerhub..."
    - docker login -u ${DOCKERHUB_USER} -p ${DOCKERHUB_PASS}
    ...
    ...
```
If use_repo_access_github_token is set to true, the environment variable REPO_ACCESS_GITHUB_TOKEN_SECRETS_ID is exposed via codebuild.

You can add this line to the beginning of your build phase commands in buildspec.yml to assign the environment variable to local variable GITHUB_TOKEN.
```
build:
    commands:
    - export GITHUB_TOKEN=${REPO_ACCESS_GITHUB_TOKEN_SECRETS_ID}
    ...
    ...
    - docker build -t $REPOSITORY_URI:latest --build-arg GITHUB_TOKEN=${GITHUB_TOKEN} .
    ...
    ...
```
AWS CloudTrail data events need to be configured in the shared service account to log S3 object-level API operations in the codepipeline buckets. The logs will be forwarded to the logging bucket in the central logging account. For example of how to log all S3 bucket object events in CloudTrail, see terraform doc here.

v1.1 Notes

If s3_block_public_access is set to true, the block public access setting for the artifact bucket is enabled.

v.1.2 Note

If use_sysdig_api_token is set to true, the secrets manager environment variable SYSDIG_API_TOKEN_SECRETS_ID is exposed via codebuild.

You can add these 8 lines to the end of your build phase commands in buildspec.yml to run Sysdig image security scans.

  build:
    commands:
      ...
      ...
      - echo "Running Sysdig image inline scan..."
      - docker run --rm -u $(id -u) -v /var/run/docker.sock:/var/run/docker.sock -v $(pwd)/reports:/staging/reports quay.io/sysdig/secure-inline-scan:2 -s https://us2.app.sysdig.com -k ${SYSDIG_API_TOKEN_SECRETS_ID} --storage-type docker-daemon --storage-path /var/run/docker.sock -r /staging/reports ${REPOSITORY_URI}:${IMAGE_TAG} || true
      - echo "Downloading Sysdig Cli Scanner..."
      - curl -LO "https://download.sysdig.com/scanning/bin/sysdig-cli-scanner/$(curl -L -s https://download.sysdig.com/scanning/sysdig-cli-scanner/latest_version.txt)/linux/amd64/sysdig-cli-scanner"
      - echo "Adding executable permission to sysdig-cli-scanner binary..."
      - chmod +x ./sysdig-cli-scanner
      - echo "Running Sysdig image cli scan..."
      - SECURE_API_TOKEN=${SYSDIG_API_TOKEN_SECRETS_ID} ./sysdig-cli-scanner --apiurl https://us2.app.sysdig.com ${REPOSITORY_URI}:${IMAGE_TAG} --policy sysdig_best_practices || true

Usage

Lambda

module "lambda_ci_pipeline" {
  source = "github.com/globeandmail/aws-ci-codepipeline?ref=1.2"

  name                                     = "app-name"
  deploy_type                              = "lambda"
  aws_organization_id                      = "aws-organization-id"
  github_repo_owner                        = "github-account-name"
  github_repo_name                         = "github-repo-name"
  github_branch_name                       = "github-branch-name"
  github_oauth_token                       = data.aws_ssm_parameter.github_token.value
  non_default_aws_provider_configurations  = {
                                               ireland = {
                                                 region_name = "region-name",
                                                 profile_name = "profile-name",
                                                 allowed_account_ids = ["account-id"]
                                               }
                                             }
  lambda_function_name                     = "lambda-function-name"
  s3_bucket_force_destroy                  = true
  create_cross_region_resources            = true
  create_ireland_region_resources          = true
  svcs_account_ireland_kms_cmk_arn_for_s3  = "svcs-account-ireland-kms-cmk-arn-for-s3"
  svcs_account_virginia_kms_cmk_arn_for_s3 = "svcs-account-virginia-kms-cmk-arn-for-s3"
  s3_block_public_access                   = true
  tags                                     = {
                                               Environment = var.environment
                                             }
}

ECS

module "ecs_ci_pipeline" {
  source = "github.com/globeandmail/aws-ci-codepipeline?ref=1.2"

  name                                         = "app-name"
  deploy_type                                  = "ecs"
  aws_organization_id                          = "aws-organization-id"
  github_repo_owner                            = "github-account-name"
  github_repo_name                             = "github-repo-name"
  github_branch_name                           = "github-branch-name"
  github_oauth_token                           = data.aws_ssm_parameter.github_token.value
  non_default_aws_provider_configurations      = {
                                                   ireland = {
                                                     region_name = "region-name",
                                                     profile_name = "profile-name",
                                                     allowed_account_ids = ["account-id"]
                                                   }
                                                 }
  s3_bucket_force_destroy                      = true
  create_cross_region_resources                = true
  create_ireland_region_resources              = true
  svcs_account_ireland_kms_cmk_arn_for_s3      = "svcs-account-ireland-kms-cmk-arn-for-s3"
  svcs_account_virginia_kms_cmk_arn_for_s3     = "svcs-account-virginia-kms-cmk-arn-for-s3"
  ecr_name                                     = "ecr-repo-name"
  use_docker_credentials                       = true
  use_repo_access_github_token                 = true
  svcs_account_github_token_aws_secret_arn     = "svcs-account-github-token-aws-secret-arn"
  svcs_account_aws_kms_cmk_arn                 = "svcs-account-aws-kms-cmk-arn"
  s3_block_public_access                       = true
  use_sysdig_api_token                         = true
  svcs_account_sysdig_api_token_aws_secret_arn = "svcs-account-sysdig-api-token-aws-secret-arn"
  tags                                         = {
                                                   Environment = var.environment
                                                 }
}

ECR

module "ecr_ci_pipeline" {
  source = "github.com/globeandmail/aws-ci-codepipeline?ref=1.2"

  name                                         = "app-name"
  deploy_type                                  = "ecr"
  aws_organization_id                          = "aws-organization-id"
  github_repo_owner                            = "github-account-name"
  github_repo_name                             = "github-repo-name"
  github_branch_name                           = "github-branch-name"
  github_oauth_token                           = data.aws_ssm_parameter.github_token.value
  non_default_aws_provider_configurations      = {
                                                   ireland = {
                                                     region_name = "region-name",
                                                     profile_name = "profile-name",
                                                     allowed_account_ids = ["account-id"]
                                                   }
                                                 }
  create_cross_region_resources                = false
  create_ireland_region_resources              = false
  svcs_account_virginia_kms_cmk_arn_for_s3     = "svcs-account-virginia-kms-cmk-arn-for-s3"
  ecr_name                                     = "ecr-repo-name"
  use_docker_credentials                       = true
  use_repo_access_github_token                 = true
  svcs_account_github_token_aws_secret_arn     = "svcs-account-github-token-aws-secret-arn"
  svcs_account_aws_kms_cmk_arn                 = "svcs-account-aws-kms-cmk-arn"
  s3_block_public_access                       = true
  use_sysdig_api_token                         = true
  svcs_account_sysdig_api_token_aws_secret_arn = "svcs-account-sysdig-api-token-aws-secret-arn"
  tags                                         = {
                                                   Environment = var.environment
                                                 }
}

Requirements

Name	Version
terraform	>= 0.12

Providers

Name	Version
aws	n/a
aws.ireland	n/a
github	n/a

Inputs

Name	Description	Type	Default	Required
aws_organization_id	(Required) The AWS organization ID.	`string`	n/a	yes
build_compute_type	(Optional) The codebuild environment compute type. Defaults to BUILD_GENERAL1_SMALL.	`string`	`"BUILD_GENERAL1_SMALL"`	no
buildspec	(Optional) The name of the buildspec file to use with codebuild. Defaults to buildspec.yml.	`string`	`"buildspec.yml"`	no
codebuild_image	(Optional) The codebuild image to use. Defaults to aws/codebuild/amazonlinux2-x86_64-standard:1.0.	`string`	`"aws/codebuild/amazonlinux2-x86_64-standard:1.0"`	no
create_cross_region_resources	(Required) Create the pipeline associated resources in all regions specified in var.non_default_aws_provider_configurations. Set to true if var.deploy_type is ecs or lambda.	`bool`	n/a	yes
create_github_webhook	(Optional) Create the github webhook that triggers codepipeline. Defaults to true.	`bool`	`true`	no
create_ireland_region_resources	(Required) Create the pipeline associated resources in the Ireland region. Set to true if var.deploy_type is ecs or lambda.	`bool`	n/a	yes
deploy_type	(Required) Must be one of the following ( ecr, ecs, lambda ).	`string`	n/a	yes
ecr_name	(Optional) The name of the ECR repo. Required if var.deploy_type is ecr or ecs.	`string`	`null`	no
github_branch_name	(Optional) The git branch name to use for the codebuild project. Defaults to master.	`string`	`"master"`	no
github_oauth_token	(Required) The GitHub oauth token.	`string`	n/a	yes
github_repo_name	(Required) The name of the GitHub repository.	`string`	n/a	yes
github_repo_owner	(Required) The owner of the GitHub repo.	`string`	n/a	yes
lambda_function_name	(Optional) The name of the lambda function to update. Required if var.deploy_type is lambda.	`string`	`null`	no
logs_retention_in_days	(Optional) Days to keep the cloudwatch logs for the codebuild project. Defaults to 14.	`number`	`14`	no
name	(Required) The name associated with the pipeline and assoicated resources. i.e.: app-name.	`string`	n/a	yes
non_default_aws_provider_configurations	(Required) A mapping of AWS provider configurations for cross-region resources creation. The configuration for Ireland region in the shared service account is required at the minimum.	map(object({ region_name = string, profile_name = string, allowed_account_ids = list(string) }))	`{}`	no
privileged_mode	(Optional) Use privileged mode for docker containers. Defaults to false.	`bool`	`false`	no
s3_block_public_access	(Optional) Enable the S3 block public access setting for the artifact bucket.	`bool`	`false`	no
s3_bucket_force_destroy	(Optional) Delete all objects in S3 bucket upon bucket deletion. S3 objects are not recoverable. Set to true if var.deploy_type is ecs or lambda. Defaults to false.	`bool`	`false`	no
svcs_account_aws_kms_cmk_arn	(Optional) The us-east-1 region AWS KMS customer managed key ARN for encrypting all AWS secrets. The key is created in the shared service account. Required if var.use_repo_access_github_token or var.use_sysdig_api_token is true.	`string`	`null`	no
svcs_account_github_token_aws_secret_arn	(Optional) The AWS secret ARN for the repo access Github token. The secret is created in the shared service account. Required if var.use_repo_access_github_token is true.	`string`	`null`	no
svcs_account_ireland_kms_cmk_arn_for_s3	(Optional) The eu-west-1 region AWS KMS customer managed key ARN for encrypting s3 data. The key is created in the shared service account. Required if var.create_ireland_region_resources is true.	`string`	`null`	no
svcs_account_sysdig_api_token_aws_secret_arn	(Optional) The AWS secret ARN for the sysdig API token. The secret is created in the shared service account. Required if var.use_sysdig_api_token is true.	`string`	`null`	no
svcs_account_virginia_kms_cmk_arn_for_s3	(Required) The us-east-1 region AWS KMS customer managed key ARN for encrypting s3 data. The key is created in the shared service account.	`string`	n/a	yes
tags	(Optional) A mapping of tags to assign to the resource	`map`	`{}`	no
use_docker_credentials	(Optional) Use dockerhub credentals stored in parameter store. Defaults to false.	`bool`	`false`	no
use_repo_access_github_token	(Optional) Allow the AWS codebuild IAM role read access to the REPO_ACCESS_GITHUB_TOKEN secrets manager secret in the shared service account. Defaults to false.	`bool`	`false`	no
use_sysdig_api_token	(Optional) Allow the AWS codebuild IAM role read access to the SYSDIG_API_TOKEN secrets manager secret in the shared service account. Defaults to false.	`bool`	`false`	no

Outputs

Name	Description
artifact_bucket_arn	n/a
artifact_bucket_id	n/a
codebuild_iam_role_name	n/a
codebuild_project_arn	n/a
codebuild_project_id	n/a
codepipeline_arn	n/a
codepipeline_id	n/a
ecr_repository_arn	n/a
ecr_repository_name	n/a
ecr_repository_url	n/a
output_artifact_object_name	n/a

Builspec examples

Lambda

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.7
  build:
    commands:
      - pip install --upgrade pip
      - pip install -r requirements.txt -t .
artifacts:
  files:
    - '**/*'
  secondary-artifacts:
    function_zip_us_east_1:
      files:
        - '**/*'
      name: lambda.zip

ECS

version: 0.2

env:
  variables:
    IMAGE_REPO_NAME: "ecr-repo-name"

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
      - AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
      - REPOSITORY_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker images...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Generating imagedefinitions.json
      - printf '[{"name":"%s","imageUri":"%s"}]' $IMAGE_REPO_NAME $REPOSITORY_URI:$IMAGE_TAG > $CODEBUILD_SRC_DIR/imagedefinitions.json

artifacts:
  files: imagedefinitions.json
  secondary-artifacts:
    imagedefinitions_file_us_east_1:
      files:
        - imagedefinitions.json
      name: imagedefinitions.zip

ECR

version: 0.2

env:
  variables:
    IMAGE_REPO_NAME: "ecr-repo-name"

phases:
  install:
    runtime-versions:
      docker: 18
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
      - AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query "Account" --output text)
      - REPOSITORY_URI=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker images...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG

globeandmail / aws-ci-codepipeline

aws-ci-codepipeline

v1.0 Notes

v1.1 Notes

v.1.2 Note

Usage

Lambda

ECS

ECR

Requirements

Providers

Inputs

Outputs

Builspec examples

Lambda

ECS

ECR

About

Languages