Simple S3 Resource for Concourse CI
Resource to upload files to S3. Unlike the the official S3 Resource, this Resource can upload or download multiple files.
Usage
Include the following in your Pipeline YAML file, replacing the values in the angle brackets (< >):
resource_types:
- name: <resource type name>
type: docker-image
source:
repository: making/s3-resource-simple
resources:
- name: <resource name>
type: <resource type name>
source:
access_key_id: {{aws-access-key}}
secret_access_key: {{aws-secret-key}}
bucket: {{aws-bucket}}
path: [<optional>, use to sync to a specific path of the bucket instead of root of bucket]
options: [<optional, see note below>]
jobs:
- name: <job name>
plan:
- <some Resource or Task that outputs files>
- put: <resource name>In case of minio,
resource_types:
- name: <resource type name>
type: docker-image
source:
repository: making/s3-resource-simple
resources:
- name: <resource name>
type: <resource type name>
source:
access_key_id: {{minio-access-key}}
secret_access_key: {{minio-secret-key}}
bucket: {{minio-bucket}}
endpoint: {{minio-endpoint-url}}
use_v4: true
path: [<optional>, use to sync to a specific path of the bucket instead of root of bucket]
options: [<optional, see note below>]
jobs:
- name: <job name>
plan:
- <some Resource or Task that outputs files>
- put: <resource name>AWS Credentials
The access_key_id and secret_access_key are optional and if not provided the EC2 Metadata service will be queried for role based credentials.
Options
The options parameter is synonymous with the options that aws cli accepts for sync. Please see S3 Sync Options and pay special attention to the Use of Exclude and Include Filters.
Given the following directory test:
test
├── results
│ ├── 1.json
│ └── 2.json
└── scripts
└── bad.sh
we can upload only the results subdirectory by using the following options in our task configuration:
options:
- "--exclude: '*'",
- "--include: 'results/*'"