Buildkite Metrics
A command-line tool for collecting Buildkite agent and job statistics for external metrics systems, focusing on enabling auto-scaling. Currently AWS Cloudwatch, StatsD and Prometheus are supported.
Installing
Either download the latest binary from Github Releases or install with:
go get github.com/buildkite/buildkite-metrics
Running
Run the metrics agent using an Agent Registration Token, found on the Buildkite Agents page:
buildkite-metrics -token abc123...
By default this will publish metrics for every queue with connected agents and unfinished jobs, and a total of all metrics. Restrict it to a single queue with -queue
if you're scaling a single cluster of agents:
buildkite-metrics -token abc123... -queue my-queue
Backends
By default metrics will be submitted to CloudWatch but the backend can be switched to StatsD or Prometheus using the command-line argument -backend statsd
or -backend prometheus
respectively.
The StatsD backend supports the following arguments
-statsd-host HOST
: The StatsD host and port (defaults to127.0.0.1:8125
).-statsd-tags
: Some StatsD servers like the agent provided by DataDog support tags. If specified, metrics will be tagged byqueue
otherwise metrics will include the queue name in the metric. Only enable this option if you know your StatsD server supports tags.
The Prometheus backend supports the following arguments
-prometheus-addr
: The local address to listen on (defaults to:8080
).-prometheus-path
: The path underprometheus-addr
to expose metrics on (defaults to/metrics
).
Upgrading from v2
- The
-org
argument is no longer needed - The
-token
argument is now an Agent Registration Token — the same used in the Buildkite Agent configuration file, and found on the Buildkite Agents page. - Build and pipeline metrics have been removed, focusing on agents and jobs by queue for auto–scaling.
If you have a compelling reason to gather build or pipeline metrics please continue to use the previous version or open an issue with details.
Development
You can build and run the binary tool locally with golang installed:
go run *.go -token [buildkite agent registration token]
Currently this will publish metrics to Cloudwatch under the custom metric prefix of Buildkite
, using AWS credentials from your environment. The machine will require the cloudwatch:PutMetricData
IAM permission.
Metrics
The following metrics are gathered when no specific queue is supplied:
Buildkite > RunningJobsCount
Buildkite > ScheduledJobsCount
Buildkite > UnfinishedJobsCount
Buildkite > IdleAgentsCount
Buildkite > BusyAgentsCount
Buildkite > TotalAgentsCount
Buildkite > (Queue) > RunningJobsCount
Buildkite > (Queue) > ScheduledJobsCount
Buildkite > (Queue) > UnfinishedJobsCount
Buildkite > (Queue) > IdleAgentsCount
Buildkite > (Queue) > BusyAgentsCount
Buildkite > (Queue) > TotalAgentsCount
When a queue is specified, only that queue's metrics are published.
AWS Lambda
An AWS Lambda bundle is created and published as part of the build process.
It's entrypoint is handler.handle
, it requires a python2.7
environment and makes use of the following env vars:
- BUILDKITE_TOKEN
- BUILDKITE_BACKEND
- BUILDKITE_QUEUE
- BUILDKITE_QUIET
Take a look at https://github.com/buildkite/elastic-ci-stack-for-aws/blob/master/templates/metrics.yml for examples of usage.
License
See LICENSE.md (MIT)