Expose secrets to your build steps. Secrets are stored encrypted-at-rest in HashiCorp Vault.
Different types of secrets are supported and exposed to your builds in appropriate ways:
ssh-agent
for SSH Private Keys- Environment Variables for strings
git-credential
via git's credential.helper
The following examples use the available authentication methods to authenticate to the Vault server, and download env secrets stored in https://my-vault-server/secret/buildkite/{pipeline}/env
and git-credentials from https://my-vault-server/secret/buildkite/{pipeline}/git-credentials
.
The keys in the env
secret are exposed in the checkout
and command
as environment variables. The git-credentials are exposed as an environment variable GIT_CONFIG_PARAMETERS
and are also exposed in the checkout
and command
.
steps:
- command: ./run_build.sh
plugins:
- vault-secrets#v1.1.0:
server: "https://my-vault-server"
path: secret/buildkite
auth:
method: "approle"
role-id: "my-role-id"
secret-env: "VAULT_SECRET_ID"
steps:
- command: ./run_build.sh
plugins:
- vault-secrets#v1.1.0:
server: "https://my-vault-server"
path: secret/buildkite
auth:
method: "aws"
aws-role-name: "my-role-name"
steps:
- command: ./run_build.sh
plugins:
- vault-secrets#v1.1.0:
server: "https://my-vault-server"
path: secret/buildkite
auth:
method: "jwt"
jwt-env: "VAULT_JWT"
It is possible to download secrets from a custom secret key, by using the secret
option on the plugin. Setting this option will tell the plugin to check the KV store for your secret (ex: secret/buildkite/supersecret
).
This secret should still follow the same conventions as the env
and environment
secrets.
steps:
- command: ./run_build.sh
plugins:
- vault-secrets#v1.1.0:
server: "https://my-vault-server"
secret: supersecret
path: secret/buildkite
auth:
method: "approle"
role-id: "my-role-id"
secret-env: "VAULT_SECRET_ID"
Secrets are downloaded by the plugin by matching the following keys, as well as the key declared in the secret
option
env
environment
private_ssh_key
id_rsa_github
git-credentials
Secrets can be uploaded to the Vault CLI, in a field called value
echo -n $(cat private_ssh_key | base64) | vault write data/buildkite/test-pipeline/private_ssh_key \
value=-
examples/
has 2 sample helper script for adding environment variables or ssh keys to Vault for a pipeline.
Environment variable secrets are handled differently in this Vault plugin to the S3 plugin.
Each environment variable is treated as an individually secret under the env
or environment
nodes for a project.
eg.
project foo/env/var1
project foo/env/var2
etc
Create policies to manage who can read and update pipeline secrets
The plugin needs at least read and list capabilities for the data. A sample read policy, this could be used by agents.
path "data/buildkite/*" {
capabilities = ["read", "list"]
}
A sample update policy for build engineers or developers. This would allow creation of secrets for pipelines, but not as defaults.
# Allow update of secrets
path "data/buildkite/*" {
capabilities = ["create", "update", "delete", "list"]
}
path "data/buildkite/env" {
capabilities = ["deny"]
}
path "data/buildkite/environment" {
capabilities = ["deny"]
}
path "data/buildkite/git-credentials" {
capabilities = ["deny"]
}
path "data/buildkite/private_ssh_key" {
capabilities = ["deny"]
}
Key values pairs can also be uploaded.
vault kv put data/buildkite/my_pipeline/environment value=- <<< $(echo "MY_SECRET=blah")
vault kv put data/buildkite/my_pipeline/env_key value=- <<< $(echo "my secret")
This example uploads an ssh key and an environment file to the base of the Vault secret path, which means it matches all pipelines that use it. You use per-pipeline overrides by adding a path prefix of /my-pipeline/
.
# generate a deploy key for your project
ssh-keygen -t rsa -b 4096 -f id_rsa_buildkite
pbcopy < id_rsa_buildkite.pub # paste this into your github deploy key
export my_pipeline=my-buildkite-secrets
echo -n $(cat id_rsa_buildkite | base64) | vault write data/buildkite/my_pipeline/private_ssh_key \
value=-
For git over https, you can use a git-credentials
file with credential urls in the format of:
https://user:password@host/path/to/repo
vault write data/buildkite/my_pipeline/git-credentials value=- <<< $(echo "https://user:password@host/path/to/repo" | base64)
These are then exposed via a gitcredential helper which will download the credentials as needed.
The Vault Secrets plugin supports a number of different configuration options.
The address of the target Vault server. Example: https://my-vault-server:8200
The key name for a custom secret. See Example
Alternative Base Path to use for Vault secrets. This is expected to be a KV Store
Defaults to: data/buildkite
Configure the Enterprise Namespace to be used when querying the vault server
Dictionary/map with the configuration of the parameters the plugin should use to authenticate with Vault.
auth
expects the following keys:
The auth method to use when authenticating with Vault. The values listed below are supported by the plugin.
Possible values:
approle
: use AppRole authentication to the Vault server (requires arole-id
be set)aws
: use AWS authentication to the Vault server (requiresaws-role-name
be set)jwt
: use JWT authentication to request a token from the Vault server
The IAM role name to be used when authenticating with AWS. If no value set, and running on an EC2 instance, defaults to the IAM role of the instance.
The role-id the plugin should use to authenticate to Vault. Has no default value
The environment variable which holds the secret-id used to authenticate to Vault. Defaults to VAULT_SECRET_ID
The environment variable which contains the JSON Web Token used to authenticate to Vault. Defaults to VAULT_JWT
Example:
steps:
- command: ./run_build.sh
plugins:
- vault-secrets#v1.1.0:
server: https://my-vault-server
auth:
method: 'approle'
role-id: 'my-role-id'
secret-env: 'MY_SECRET_ENV'
The unit tests are written using BATS, you can test locally with:
make test
or using docker-compose:
docker-compose -f docker-compose.yml run --rm tests
The integration tests are run by spinning up a local vault container in dev mode, and configuring them with some data.
make integration-test
When writing test plans, note that secrets are processed in the order they appear in the list returned from the Vault.
You can test the pipeline locally using the bk cli
. Passing the -E BUILDKITE_PLUGIN_DOCKER_COMPOSE_RUN_LABELS=false
value will prevent the docker-compose plugin
from trying to use variables that don't exist when running the pipeline locally.
bk local run -E BUILDKITE_PLUGIN_DOCKER_COMPOSE_RUN_LABELS=false
A special thank you to the original author @mikeknox for providing the framework for this plugin
MIT (see LICENSE)