tonellotto / terrier-micro

An efficient layer to perform query processing on top of Terrier

Terrier Micro

This project provides a lightweight implementation of some query processing strategies built on top of Terrier 5. It re-implements the query processing pipeline of the Terrier search engine, removing all unnecessary features such as document score modifiers, multiple weighting models, etc.

If you use this package to conduct search or experimentation, whether be it a research paper, dissertation, article, poster, presentation, or documentation, please cite the following paper:

@article{fnt,
    author = {Tonellotto, Nicola and Macdonald, Craig and Ounis, Iadh},
    issn = {1554-0669},
    journal = {Foundations and Trends in Information Retrieval},
    number = {4--5},
    pages = {319--492},
    title = {Efficient Query Processing for Scalable Web Search},
    volume = {12},
    year = {2018}
}

This package is free software distributed under the GNU Lesser General Public License.

Pre-requisites

Elias-Fano compression for Terrier is required for testing purposes or if you plan to use it in your experiments, but it is not explicitly required for using the Terrier Micro package.

To install the Elias-Fano compression for Terrier package (version 1.5.1) on your local machine, please run the following commands.

git clone https://github.com/tonellotto/terrier-ef
cd terrier-ef
git checkout 1.5.1
mvn install appassembler:assemble

Usage

If not already available, e.g. from Maven Central, you should git clone and install Terrier Micro (version 1.5.1):

git clone https://github.com/tonellotto/terrier-micro
cd terrier-micro
git checkout 1.5.1
mvn install appassembler:assemble

The main script to perform batch query processing is the retrieve tool.

If you want to use all available processors on your machine to perform batch query processing, use the parallel retrieve tool.

Two other scripts are provided, to support advanced query processing strategies: the ms-generate and bmw-generate tools.

Python

The python folder repo holds static copies of notebooks for learning to use Terrier Micro (Java). The notebooks in this repo are sync'ed (by hand) with notebooks in Colab. For convenience, there is a small pre-built index, available to download here.

• Terrier Micro demo on Robust 2004: local and colab notebooks.

Credits

Developed by Nicola Tonellotto, ISTI-CNR. Contributions by Craig Macdonald, University of Glasgow, and Matteo Catena, ISTI-CNR.