Test Summary Buildkite Plugin

A Buildkite plugin that adds a single annotation for all your test failures using buildkite-agent annotate.

Supported formats:

JUnit
TAP^
Plain text files with one failure per line

^ Current TAP support is fairly limited. If you have an example TAP file that is not being interpreted correctly, feel free to open an issue or pull request.

Example

Upload test results as artifacts using any supported format. If you include the $BUILDKITE_JOB_ID in the path, a link to the build will be included in the annotation. Some examples:

steps:
  - label: rspec
    command: rspec
    parallelism: 10
    # With spec_helper.rb:
    # RSpec.configure do |config|
    #   config.add_formatter('RspecJunitFormatter', "artifacts/rspec-#{ENV['BUILDKITE_JOB_ID']}.xml")
    # end
    artifact_paths: "artifacts/*"

  - label: ava
    command: bash -c "yarn --silent test --tap > artifacts/ava.tap"
    artifact_paths: "artifacts/*"

  - label: rubocop
    # The emacs format is plain text with one failure per line
    command: rubocop -f emacs -o artifacts/rubocop.txt
    artifact_paths: "artifacts/*"

Wait for all the tests to finish:

  - wait: ~
    continue_on_failure: true

Add a build step using the test-summary plugin:

  - label: annotate
    plugins:
      bugcrowd/test-summary#v1.3.0:
        inputs:
          - label: rspec
            artifact_path: artifacts/rspec*
            type: junit
          - label: ava
            artifact_path: artifacts/ava.tap
            type: tap
          - label: rubocop
            artifact_path: artifacts/rubocop.txt
            type: oneline
        formatter:
          type: details
        context: test-summary

See buildkite annotation of all the failures. There are some example annotations included below.

Configuration

Inputs

The plugin takes a list of input sources. Each input source has:

label: the name used in the heading to identify the test group.
artifact_path: a glob used to download one or more artifacts.
type: one of junit, tap or oneline.
encoding: The file encoding to use. Defaults to UTF-8.
strip_colors: Remove ANSI color escape sequences. Defaults to false.
crop: (oneline type only) Number of lines to crop from the start and end of the file, to get around default headers and footers. Eg:

crop:
  start: 3
  end: 2

job_id_regex: Ruby regular expression to extract the job_id from the artifact path. It must contain a named capture with the name job_id. Defaults to (?<job_id>[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}).

Formatter

There are two formatter types, summary and details.

The summary formatter includes a single line for each failure.

The details formatter includes extra information about the failure in an accordion (if available). This is the default option.

Other formatter options are:

show_first: The number of failures to show before hiding the rest inside an accordion. If set to zero, all failures will be hidden by default. If set to a negative number, all failures will be shown. Defaults to 20.

Note: If the generated annotation exceeds the maximum allowed size, it will fallback from the details formatter to the summary formatter and then to only displaying the failure count for each input source.

Other options

context: The Buildkite annotation context. Defaults to test-summary.
style: Set the annotation style. Defaults to error.
fail_on_error: Whether the command should return non-zero exit status on failure. Defaults to false so failing to annotate a build does not cause the entire pipeline to fail.

Developing

To run the tests:

docker-compose run --rm test rspec

If you have ruby set up, you can just run:

bundle install
rspec

To generate sample markdown based on the files in spec/sample_artifacts:

bin/run-dev

Release process

Update version.rb
Update version in README example
Update CHANGELOG.md
Ensure screenshots are up to date
Push to github and ensure tests pass
docker build -t tessereth/test-summary-buildkite-plugin:vx.x.x .
git tag --sign vx.x.x -m "Release vx.x.x"
docker push tessereth/test-summary-buildkite-plugin:vx.x.x
git push origin vx.x.x
Copy changelog entry to github release notes

faun / test-summary-buildkite-plugin