Incorrect P@5 Values
ryan-clancy opened this issue · comments
When doing the official run, I get the following values from trec_eval:
Starting container from saved image...
Logs for search in container with ID eade5686b363406fb16c3d8c51b37436cd1879429c53ee11ba0cb60cae33f752...
CREATE DATABASE
CREATE CONNECTION
SCORING TOPICS
Evaluating results using trec_eval...
###
# /tmp/output/olddog/run.bm25.robust04
###
map all 0.1771
P_5 all 0.4385
MAP looks good, but the P@5 value isn't what's expected.
Log and run files available at https://github.com/osirrc/osirrc2019-runs/tree/master/olddog/robust04/2019-06-17
I am checking right now, but I think that the measure in the Table is P@30 rather than P@5, so our mistake in which number to report.
Confirmed - the number we put in the table is P@30.
Will be fixed!
@r-clancy and @lintool a question:
I computed P@5 using trec_eval and get the value @r-clancy reports above (Phew);
but if I use trec_eval -c, I get P@5 = 0.4297.
I think people should report the latter, not the former - i.e., a system has to be evaluated on all topics, not on all topics that they returned answers for. (Otherwise, I could get a very high MAP by being smart about selecting topics to include - even if that is difficult to do, it is not unimaginable.)
@arjenpdevries filed issue here: osirrc/jig#105