More bootstraps and bigger confidence interval support
tomato42 opened this issue · comments
Feature request
Is your feature request related to a problem? Please describe
The analysis.py script provides just one confidence interval: 95%. While that is related to the 2-sigma interval of normal distribution, so in theory allows for easy estimation of bigger confidence intervals (like 3, 4, 5-sigma), it assumes a normal distribution, which the differences don't follow, and may be very different than the Student t distribution too. So it would be nice if those confidence intervals were configurable.
Describe the solution you'd like
The analysis.py should accept an argument that specifies the user-requested CI.
As we use bootstrapping for calculating current CI, we will need to increase the number of bootstraps for bigger CIs.
Given that the number of repeats influences the confidence in the given value, we should allow setting the number
of bootstrap repeats too.
Describe alternatives you've considered
While it can be calculated externally with R, it looks like scipy is faster for doing this.