[Feature] Outlier clients filteration

[Dmitry Balabka]

Description of Feature

Usually, during experiments, we can face customers who perform enormous amounts of transactions. Such outliers lead to bias and can change experiment results. To fix this issue, we have filtered out such clients based on the total amount of transactions during the experiment. Currently, Growthbook supports capping for revenue and count. Unfortunately, it does not work for binomial metrics.

I tried to use count metric with capping with pretty simple binomial metric SQL adjustments:

SELECT
  e.value.string_value as ecentria_id,
  user_pseudo_id,
  TIMESTAMP_MICROS(event_timestamp) as timestamp,
  1 as value  -- <--------- this line was added
FROM
....

However, capping does not work as expected. It filters out rows that have a value higher than the specific threshold. While I'm looking into filtering out all rows that have a relation to the client that contributes an enormous amount of transactions.

Having a feature that helps filter out clients based on a threshold (absolute or percentile) by aggregated values over all time of the experiment, can help to increase test sensitivity by decreasing variation.

We definitely can workaround it with custom SQL, but unfortunately, it requires hardcoded logic in each metric.

Related Issues

TBD