Tests: support using a matrix for testing
fahad19 opened this issue · comments
Currently
Tests specs can be written for features and segments: https://featurevisor.com/docs/testing/
Example test spec looks like this:
# tests/foo.feature.yml
feature: foo
assertions:
- description: Testing variation at 40% in NL
environment: production
at: 40
context:
country: nl
expectedToBeEnabled: true
Issue
For testing various scenarios, we have to write an assertion manually and this can lead to a lot of manual effort.
Proposal
Introduce a matrix, similar to GitHub Actions: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs
Those who wish, can benefit from defining the matrix in their assertions:
feature: foo
assertions:
- matrix:
at: [40]
version: ["1.0", "2.0", "3.0"]
country: ["nl", "de", "us"]
description: "Testing version ${{ version }} in ${{ country }}"
at: ${{ at }}
context:
version: ${{ version }}
country: ${{ country }}
expectedToBeEnabled: true
With above spec, the assertion will then run against all the combinations of version
and country