Split doesn't proportionally split traffic across utm_source, utm_medium and utm_campaign.
PrashannaGuru opened this issue · comments
Describe the bug
The Split tool doesn't proportionally split traffic across utm_source, utm_medium and utm_campaign which requires us to normalize the data for traffic and calculate the statistical significance outside of split.
To Reproduce
We see the page visits in Mixpanel and the number of page views to each variant from different utm_source, utm_medium and utm_campaign is not the same.
Expected behavior
If the traffic is split 50:50, then each variant should get an equal number of page views from all the campaigns that bring users to the page. Therefore, we can be confident of the conversion difference and statistical significance in the Split dashboard.
By default, split uses Split::Algorithms::WeightedSample to select the alternative for a given user. That particular algorithm does not use any external params to do that. https://github.com/splitrb/split#algorithms
With some changes, it would be possible to write a custom algorithm for it. But maybe a workaround would be to define a different experiment for each utm_*?
As Split does not use any utm_params by default I'm closing this for now.