Support creating metalink files

[Danny Berger]

Currently, a previous task must create the metalink file before this can be resource can be used. Often, that task runs the same sort of import-file/file-upload commands. This gets redundant across multiple pipelines and repositories, and can probably be made simpler.

Alternatively, allow put operations to receive a list of files from which to generate a metalink using import-file. For the file-upload, it should probably receive a list of upload targets. Currently upload configuration can only be specified by environment variables.

A future put/out interface might look like...

• metalink - path to the metalink file (one of metalink or files must be specified)
• files - an array of file glob paths to create a new metalink file from (one of metalink or files must be specified)
• upload_files - an array of hashes to upload files when creating a metalink with files
  • to - a supported upload path with basic template interpolation support (e.g. s3://hostname/bucketname/{{.Name}}; version is available with {{.Version}}, and checksums are available, e.g. {{.SHA1}})
  • env - a hash of configuration to specify through environment variables for the upload (e.g. AWS_ACCESS_KEY_ID)
• version_file - a path to a file whose contents have a version number to include when creating a metalink with files
• rename - publish the metalink file with a different file name (only applicable with metalink)
• options - a hash of supported options, depending on the repository type
  • for git repositories
    • author_name, author_email - the commit author
    • message - the commit message

With an example pipeline usage being...

jobs:
- name: build
  plan:
  - get: repo
  - get: version
    params:
      bump: rc
  - task: build
      file: repo/ci/tasks/build/config.yml
    - put: repo-dev-artifacts
      params:
        files:
        - ssoca-*
        version_file: version/version
        upload_files:
        - to: "s3://s3-external-1.amazonaws.com/my-release/releases/{{.Version}}/{{.Name}}"
          env:
            AWS_ACCESS_KEY_ID: ((assets_s3_access_key))
            AWS_SECRET_ACCESS_KEY: ((assets_s3_secret_key))
resources:
- name: repo-dev-artifacts
  type: metalink-repository
  source:
    uri: git+ssh://[e-mail]:my/release.git//dev#artifacts
    options:
      private_key: ((git_private_key))

This does result in a fairly complex put configuration, though. In theory, the upload_files could (should?) become part of the resource configuration and be renamed to upload_options/upload/mirrors? This would help ensure consistency across puts to the resource, and reduce the amount of configuration needed for resource operations. If termed mirrors, it would also allow mirroring existing metalinks to other targets as they are being added to a repository.

Perhaps something more like...

jobs:
- name: build
  plan:
  - get: repo
  - get: version
    params:
      bump: rc
  - task: build
      file: repo/ci/tasks/build/config.yml
    - put: repo-dev-artifacts
      params:
        files:
        - ssoca-*
        version_file: version/version
resources:
- name: repo-dev-artifacts
  type: metalink-repository
  source:
    uri: git+ssh://[e-mail]:my/release.git//dev#artifacts
    options:
      private_key: ((git_private_key))
    mirrors:
    - uri: "s3://s3-external-1.amazonaws.com/my-release/dev/{{.Version}}/{{.Name}}"
      env:
        AWS_ACCESS_KEY_ID: ((assets_s3_access_key))
        AWS_SECRET_ACCESS_KEY: ((assets_s3_secret_key))

[Danny Berger]

Implemented via d05e7d0.