MobileTeleSystems / Ambrosia

Ambrosia is a Python library for A/B tests design, split and result measurement

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Multiple comparisons problem corrections

xandaau opened this issue · comments

In the Tester class when one use multiple experimantal groups or several metrics only Bonferroni correction is supported.

It will be useful to implement some more complex and popular classic corrections for MCP (Holm, Benjamini–Hochberg, etc).
It should be noted that current structure of `Tester`` may not be so convenient to add these corrections, so it will need to change the main class code.
These corrections should be discussed before the implementation as well as the problem of correct confidence intervals calculation.