Inconsistent numerical results

Question

Inconsistent numerical results

gdalle opened this issue a year ago · comments

The numerical results reported in the JOSS paper are inconsistend with the latest CI benchmark https://github.com/rikhuijzer/SIRUS.jl/actions/runs/5972815140

openjournals/joss-reviews#5786

Guillaume Dalle commented a year ago

lgtm

Rik Huijzer · Answer 1 · Wed Sep 13 2023 19:23:33 GMT+0800 (China Standard Time)

Good catch. One problem was that I hadn't set the RNG for all DecisionTree runs. That was fixed in 8153378. Furthermore, I've tried to improve the inconsistencies between the CI runs in #53, but this wasn't successful. The most likely culprit at this point seems inconsistencies between hardware in the CI runs. Since results between runs on the same system are deterministic and since one issue is with DecisionTree.jl, I hope we can agree that consistency is good enough for now. Otherwise, feel free to reopen.

I've updated the source for the paper in 632d6b1. Apart from the $0.69 \pm 0.09$ to $0.77 \pm 0.07$ jump for the Iris dataset, not much has changed (source: https://github.com/rikhuijzer/SIRUS.jl/actions/runs/6171051743).