S3 Cache Buildkite Plugin

Save and restore cache to and from AWS S3.

Example

Add the following to your pipeline.yml:

steps:
  - command: npm install && npm test
    plugins:
      - peakon/s3-cache#v2.2.1:
          id: CACHE_IDENTIFIER # optional, default: none
          aws_profile: aws-profile-name # optional, default: none
          restore_dry_run: false # set it to "true" to only check if cacheKey is present on S3 (no download / restoring)
          save:
            - key: 'v1-node-modules-{{ checksum("package-lock.json") }}' # required
              paths: [ "node_modules" ] # required, array of strings
              when: on_success # optional, one of {always, on_success, on_failure}, default: on_success
              overwrite: false # optional, set true to overwrite cache on S3 even if object already exists
          restore:
            - keys:
                - 'v1-node-modules-{{ checksum "package-lock.json" }}'
                - 'v1-node-modules-' # will load latest cache starting with v1-node-modules- (not yet implemented)

Configuration

Prerequisites

Make sure to set BUILDKITE_PLUGIN_S3_CACHE_BUCKET_NAME=your-cache-bucket-name before using this plugin.

Plugin

You can specify either save or restore or both of them for a single pipeline step.

Checking if cache was successfully restored

In some cases you may need to build a conditional logic in the build command based on the results of cache restore operation (for example, to avoid re-generating the cache which already exists and was restored successfully).

To support this use-case, this plugin exports environment variables that can be used during a command step. The feature is opt-in and requires id to be specified in plugin configuration.

For example, this step generates a cache of node_modules (which is then used by all jobs that need it):

steps:
  - command: "[ ! \"${BUILDKITE_PLUGIN_S3_CACHE_npm_0_KEY_0_HIT}\" =~ ^(true)$ ] && npm install"
    plugins:
      - peakon/s3-cache#2.2.1:
          id: npm
          restore_dry_run: true # This saves runtime, but doesn't check for integrity 
          restore:
            - keys: [ 'v1-node-modules-{{ checksum "package-lock.json" }}' ]
          save:
            - key: 'v1-node-modules-{{ checksum "package-lock.json" }}'
              paths: [ "node_modules" ]

Supported functions

checksum 'filename' - sha256 hash of a filename
epoch - time in seconds since Unix epoch (in UTC)
.Environment.SOME_VAR - a value of environment variable SOME_VAR

AWS profiles

You can specify a custom AWS profile to be used by AWS CLI

in pipeline YAML (aws_profile: profile_name)
via BUILDKITE_PLUGIN_S3_CACHE_AWS_PROFILE environment variable (e.g. inside agent environment hook).

Developing

To run the tests:

docker-compose run --rm tests

Contributing

Fork the repo
Make the changes
Run the tests
Commit and push your changes
Send a pull request

toastercup / s3-cache-buildkite-plugin