Several for-loops in evaluation don't seem to do anything
amueller opened this issue · comments
There's some for-loops in the evaluation that only work if they run exactly once, which seems pretty confusing.
One is in https://github.com/automl/TabPFN/blob/main/tabpfn/RunFullDatasetAnalyses.ipynb, in "eval_methods".
If there's more than one did, it only returns the results of the last iteration. The code always calls it with only one did, so that's fine, but then maybe remove the for-loop?
The other one is here:
https://github.com/automl/TabPFN/blob/main/tabpfn/scripts/baseline_prediction_interface.py
That one has a TQDM attached, but it returns in the first iteration. Again, I think it's only called with lists of length one, but maybe then removing the loop would be better?
Btw, I rewrote eval_methods to take datasets, not "dids" since the dids rely on global state about how things are numbered and add some extra indirection. If you want a PR for the refactor, let me know.
I know this is all very sub-optimally written for people working on the code now, as this is very much still the code in which we tried out a lot of inference tricks just before the deadline. Sorry for this.
Internally we moved on to a new implementation of this that is a bit cleaner. I think we will share that with the next publication.
That's fair, though from what Frank said, that's maybe still a bit out? By that time I might have to publish my own code for a submission, so then we'll have three different implementations. That doesn't seem ideal.