The control group is not always the default in RSpec tests
dorianmariecom opened this issue · comments
Describe the bug
I have a system test / integration test that shows an alternative version instead of the control one
To Reproduce
Have an ab_test
Expected behavior
Have control versions in tests by default
Additional context
Weird, it seems like it was the case, not sure what happened, maybe on my side
Hi @dorianmariefr 🙌
Right now, Split doesn't know about test environments. So, it should be working as it should in production envs. Is it randomized or does it always returns the alternate version?
As 'control' is the alternative returned when split is disabled, another way to provide this behavior is to disable Split on tests:
Split.configuration.enabled = false
The ab_test call will also return 'control' when checking User-Agent for Bots or IP Filters, etc.. Maybe you had something like this on CI and it was lost? 🤔
I do believe we need some better syntax/support to declare which alternatives split should return for test envs, and be able to test any version of a given experiment easily :)
As the current workaround is to mock ab_testmethod directly to return the alternatives you want.
thanks ❤️ works well to disable Split in tests, didn't mock it yet, but maybe in the future