A simple package to download and standardize TREC experiment data. It generates or includes,
- standardized title, description, and narrative queries
- qrels for the full corpus and popular "news-only" subsets
- standard stopword list (indri)
The build process will download and process qrels from NIST and other servers for the following datasets,
- trec12: topics 51-200 associated with all documents on Tipster Disks 1 and 2.
- trec12-news: topics 51-200 associated with only news documents on Tipster Disks 1 and 2. This only includes the AP, WSJ, and Ziff-Davis documents in the qrels.
- trec45: the Robust 2004 topics associated with documents on TREC Disks 4 and 5 except the Congressional Record.
- trec45-news: the Robust 2004 topics associated with only news documents on TREC Disks 4 and 5. This only includes the FBIS, FT, and LA Times documents in the qrels.
- nyt: the Common Core 2017 topics associated with all documents in the New York Times Annotated Corpus.
- msmarco: the topics associated with documents in the MS MARCO dataset.
- mq: 60k unjudged queries associated with the Million Query Track.
- aol: ~7.5M unique unjudged queries associated with a filtered version of the AOL Query Log. Please be prepared to deal with the ethical issues raised in using this dataset.
For each set of queries (i.e. trec12, trec45, and nyt), we generate title, description, and narrative queries. The MS Marco dataset only has title queries.
qrels/ relevance judgements for test collections
queries/ queries for tests collections
qlogs/ queries with no associated relevance judgments
misc/ miscellaneous data for experiments
make
@online{diaz:trec-data,
author = {Diaz, Fernando},
title = {trec-data},
year = {2018},
url = {https://github.com/diazf/trec-data}
}
Thanks to Hamed Zamani and Mostafa Dehghani for help with AOL processing logic.