Can one calculate the critical distance value based on your implementation of Holm's post hoc test?
NinaOmejc opened this issue · comments
Hello,
first, thanks for the open code!
I was wondering whether there's a way to calculate critical distance value as is calculated in the original implementation in Orange (by Demšar), where the Nemenyi or Bonferroni-Dunn test was used. Since your implementation only includes adjusted p-values, I assume it is not possible, as Holm's method doesn't have a table of critical values? (The only option is to visually inspect the diagram if that even makes sense?)
Thanks!
Hello,
If I understood correctly, you want to get the statistical value (critical difference) instead of the p-value only. This can be done simply by fetching the z-statistic value of the Wilcoxon Signed Rank Test from the output list.
In this code for the cd-diagram, only the p-value is extracted in order to show the significance in difference of performance, but one can extract as well the z-statistics scipy-stats-wilcoxon [docs]
You can see in the code at the author did the following:
p_value = wilcoxon(perf_1, perf_2, zero_method='pratt')[1]
By replacing [1] by [0] than one can get the z-statistic value. So you can add another line to extract it and save it if you need it for more analysis on the cd-diagram.
Thanks for the quick reply!
Hm, yes, to some extent z-scores are helpful but I am more interested in how to find the critical value of the statistical test in general and not related to individual samples. For example, the critical value of the two-sided Z statistic distribution with alpha=0.05, equals -+ 1.96, so it's a fixed number, regardless of the values of my samples (or their individual z-statistic values).
In Orange, this is calculated using the function "compute_cd", which depends on the test used, number of datasets (N) and classifiers (k) but is not dependent on specific values inside the dataset:
https://github.com/janezd/orange3/blob/22f5f91c4e515fa4d521dcdf756f7d842698b7ad/Orange/evaluation/scoring.py#L373
But I'm still wondering whether it's even possible to do it in your implementation, as the alpha changes and gets adjusted in between when this Holm adjustment is used.
The thing is I was asked about this critical value and I couldn't find it quantitatively. I can somehow find it graphically though, as it should be somewhere between the distance of the largest thick (nonsignificant) line and that distance plus the distance to the rank of the nearest significantly different method.
Oh okay, well then in that case, I think the critical value table of the Wilcoxon Signed Ranked Test can be added in the code in order to use it as in the Orange library.
Yeah I tried that but unfortunately, the result does not make sense (it's too big). I'll think about it a bit more, whether I can somehow extract it from adjusted p-values. Thanks for the discussion! :)