Precision & Coverage Issue, Possible effects on anchor?
GDPlumb opened this issue · comments
I was running playing around with Anchors and was getting really weird coverage and precision values. Then I saw the TODO (line 301 of anchor_tabular.py) saying that the precision and coverage measures are incorrect.
Does this mean that the anchors that I computed were also wrong? Or does the issue only affect those metrics?
Thanks!
That is a note on the precision and coverage of intermediate anchors, especially for when features are ordinal or numerical. Are you seeing weird coverage / precision values on the final anchors produced?
All of my features are numerical.
I'm getting coverage (either 0 or 1) & precision (doesn't make sense for the data generation process) values that are unusual for the final anchors. The final anchor also sometimes doesn't apply to the point it was generated for.
Unfortunately I didn't test the numerical stuff enough to be as confident about it as discretized data. That definitely should never happen (that an anchor does not apply to the point it was generated for). Could you share an example by any chance?
Here it is (should be .py)
Is the way I set up the code incorrect? If not, what would you suggest doing?
Thanks!
Sorry for the long delay in responding to this. The way you setup the code is not incorrect, there was in fact an error in how I was sampling numerical features that get discretized.
I fixed it and updated the package, would you mind trying again? Please reopen the issue if it is not fixed.
For your example, quartiles are not enough to get decent anchors, so you may want to try to set discretizer=decile in the fit function. Even if you don't, you shouldn't see anchors that do not apply to the points they were generated for, and precision / coverage should be right.