featurevisor / featurevisor

Feature flags, experiments, and remote config management with GitOps

Home Page:https://featurevisor.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Tests: support using a matrix for testing

fahad19 opened this issue · comments

Currently

Tests specs can be written for features and segments: https://featurevisor.com/docs/testing/

Example test spec looks like this:

# tests/foo.feature.yml
feature: foo

assertions:
  - description: Testing variation at 40% in NL
    environment: production
    at: 40
    context:
      country: nl
    expectedToBeEnabled: true

Issue

For testing various scenarios, we have to write an assertion manually and this can lead to a lot of manual effort.

Proposal

Introduce a matrix, similar to GitHub Actions: https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs

Those who wish, can benefit from defining the matrix in their assertions:

feature: foo

assertions:
  - matrix:
      at: [40]
      version: ["1.0", "2.0", "3.0"]
      country: ["nl", "de", "us"]
    description: "Testing version ${{ version }} in ${{ country }}"
    at: ${{ at }}
    context:
      version: ${{ version }}
      country: ${{ country }}
    expectedToBeEnabled: true

With above spec, the assertion will then run against all the combinations of version and country