With the synthetics-test-automation-bitrise-step-run-tests step, you can run Synthetics tests during your Bitrise CI and ensure all your teams using Bitrise can benefit from Synthetic tests at every stage of the software lifecycle. This step uses the Datadog CI Synthetics command.
To get started:
- Add this step to your workflow. You can see Bitrise's documentation on how to do that. You can also configure it locally by referencing this step in your
bitrise.yml. - Add your API and App keys to your secrets in Bitrise. Documentation on how to do that can be found here: Setting a Secret.
- Configure your step inputs (see the Step Inputs documentation). You can also configure them in your
bitrise.yml. The only required inputs are the two secrets you configured earlier. The rest of the possible inputs are described in a later section.
This Step can be run directly with the Bitrise CLI.
To get started:
- Open up your Terminal / Command Line
git clonethe repositorycdinto the directory of the step (the one you justgit cloned)- Create a
.bitrise.secrets.ymlfile in the same directory ofbitrise.yml(the.bitrise.secrets.ymlis a git ignored file, you can store your secrets in it) - Check the
bitrise.ymlfile for any secret you should set in.bitrise.secrets.yml - Once you have the required secret parameters in your
.bitrise.secrets.yml, run this step with the Bitrise CLI:bitrise run test.
An example .bitrise.secrets.yml file:
envs:
- A_SECRET_PARAM_ONE: the value for secret one
- A_SECRET_PARAM_TWO: the value for secret two- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- files: 'e2e-tests/*.synthetics.json'For an example test file, see this test.synthetics.json file.
- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- test_search_query: 'tag:e2e-tests'- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- test_search_query: 'tag:e2e-tests'
- variables: |
START_URL=https://staging.website.com
PASSWORD=$STAGING_PASSWORDThis task overrides the path to the global datadog-ci.config.json file.
- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- config_path: './synthetics-config.json'For an example configuration file, see the global.config.json file.
For reference here's how a full configuration could look:
- datadog-mobile-app-run-tests@1:
inputs:
- api_key: $DATADOG_API_KEY
- app_key: $DATADOG_APP_KEY
- config_path: './global.config.json'
- device_ids: 'apple iphone se (2022),15.4.1, apple iphone 14 pro,16.1'
- fail_on_critical_errors: true
- fail_on_missing_tests: true
- fail_on_timeout: true
- files: 'e2e-tests/*.synthetics.json'
- junit_report: 'e2e-test-junit'
- locations: 'aws:us-west-1'
- mobile_application_version: '01234567-8888-9999-abcd-efffffffffff'
- mobile_application_version_file_path: 'path/to/application.apk'
- polling_timeout: 4200000
- public_ids: 'abc-d3f-ghi, jkl-mn0-pqr'
- site: 'datadoghq.com'
- subdomain: 'myorg'
- test_search_query: 'tag:e2e-tests'
- tunnel: true
- variables: |
START_URL=https://staging.website.com
PASSWORD=$STAGING_PASSWORD| Name | Requirement | Description |
|---|---|---|
apiKey |
required | Your Datadog API key. This key is created by your Datadog organization and will be accessed as an environment variable. |
appKey |
required | Your Datadog application key. This key is created by your Datadog organization and will be accessed as an environment variable. |
configPath |
optional | The global JSON configuration is used when launching tests. See the example configuration for more details. |
deviceIds |
optional | Override the mobile device(s) to run your mobile test. |
locations |
optional | String of locations separated by semicolons to override the locations where your tests run. |
failOnCriticalErrors |
optional | A boolean flag that fails the CI job if no tests were triggered, or results could not be fetched from Datadog. The default is set to false. |
failOnMissingTests |
optional | Fail the CI job if at least one specified test with a public ID (using publicIds or listed in a test file) is missing in a run (for example, if it has been deleted programmatically or on the Datadog site). |
failOnTimeout |
optional | A boolean flag that fails the CI job if at least one test exceeds the default test timeout. The default is set to true. |
files |
optional | Glob patterns to detect Synthetic test configuration files. |
jUnitReport |
optional | The filename for a JUnit report if you want to generate one. |
mobileApplicationVersion |
optional | Override the default mobile application version for a Synthetic mobile application test. The version must be uploaded and available within Datadog. This version is also outputted by the datadog-mobile-app-upload step. |
mobileApplicationVersionFilePath |
optional | Override the application version for Synthetic mobile application tests. |
pollingTimeout |
optional | The duration (in milliseconds) after which the batch is deemed as failed because of a timeout. The default is 30 minutes. |
publicIds |
optional | String of public IDs separated by commas for Synthetic tests you want to trigger. |
site |
optional | The Datadog site to send data to. If the DD_SITE environment variable is set, it takes precedence. |
subdomain |
optional | The name of the custom subdomain set to access your Datadog application. If the URL used to access Datadog is myorg.datadoghq.com, the subdomain value needs to be set to myorg. |
testSearchQuery |
optional | Trigger tests corresponding to a search query. This can be useful if you are tagging your test configurations. See best practices for more information on tagging. |
tunnel |
optional | Enable tunnel to interact with Datadog API. |
variables |
optional | Key-value pairs for injecting variables into tests. Must be formatted using KEY=VALUE. |
Additional helpful documentation, links, and articles: