Separate experiment administration from scenario definition
wadim opened this issue · comments
Experiment administration and scenario definition are fundamentally separate concerns.
Experiment administration is concerned with issues such as:
- which experiments are running at the moment
- sample size (number of people to expose to the experiment, time frame of experiment)
- configuration of random assignment of subjects to groups: how many subjects to each group, what randomization to use
Scenario definition is only concerned with what each scenario looks like.
These concerns are fundamentally disconnected. They call for different workflows and they can be prepared by different groups of people.
Ably should not rely on any specific method of test administration, specifically on any specific way of passing test definitions to Ably. We need to separate defining tests from describing variants and define a minimal interface between them:
the only thing the describing variants module needs from the defining tests module is the information which variant of each test is the selected one.