aws / aws-cdk-rfcs

RFCs for the AWS CDK

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Release public artifacts (lambda layers for custom resources, docker images)

eladb opened this issue · comments

PR Champion
#

Description

When the AWS construct library includes an asset such as custom resource providers, lambda bundles, docker images or static files, we currently bundle these into the library and then build and publish it to user accounts. Since the CDK is public and open-source, it should be possible to publish these assets to S3/ECR in all regions and consume them as public assets. This will reduce the size of our libraries and the time it takes for these constructs to be deployed.

Progress

  • Tracking Issue Created
  • RFC PR Created
  • Core Team Member Assigned
  • Initial Approval / Final Comment Period
  • Ready For Implementation
    • implementation issue 1
  • Resolved

Let's focus on S3 for Lambda bundles because that is a pertinent use case we have to reduce the side of the mono-module. Specifically, the aws-s3-deployment module currently needs to bundle the entire AWS CLI because it leverages s3 sync. Another use case we already have is the aws-eks which currently uses an open source AWS Lambda Layer published as a SAR app which contains kubectl, helm and the AWS CLI.

The only value of using SAR here would be that they take care of replication but that's not a problem with S3 since S3 now supports cross-region replication.

So the design approach would be to create a public S3 bucket in each region we support with a well-known name and cross-region replication. We will store .zip files for AWS Lambda layers there which will be mounted into custom resources of AWS constructs such as s3-deployment and EKS.

To achieve this, we will add another type of artifact to our build system (e.g. add dist/s3 or something like that) and during publishing we will upload all the files from this directory to the replicated S3 bucket (with a key prefix that denotes the CDK version (e.g. s3://aws-cdk-files-eu-west-2/v1.22.0/awscli-layer.zip). We can also decide to just upload it to all the regional buckets. Shouldn't be too problematic and might be better because it will give us a synchronous confirmation that we are done.

Note that this is susceptible to an eventual consistency issue with the publishing of the actual modules that reference the artifact, so perhaps it has to be the first publisher (i.e. a separate pipeline stage or run-order).

The publisher will no-op if there is already a file in that location, which is aligned with how we publish (publishing only really happens when we actually bump the version and published artifacts are immutable).

Agree. And let's include cn-north-1 and cn-northwest-1 S3 buckets in the pipeline so we can roll out the artifacts into AWS China regions.

Let's also include us-gov-west-1 and us-gov-east-1 to cover our GovCloud customers.

Exactly for those reasons (no S3 replicating to non-commercial partitions, and keeping up with new region launches) -- that it's going to be a lot of work keeping up with users in all regions.

We would be opting in to a somewhat large operational burden, and I don't really see the advantages outweighing that.

I think we don’t have to do this as part of 2.0, but I suspect that the size of monocdk will eventually be a motivating factor to implement this.

I agree it’s not trivial.