dli1 / auto-stop-tar

When to Stop Reviewing in Technology-Assisted Reviews

auto-stop-tar

Code and data for paper "When to Stop Reviewing in Technology-Assisted Reviews"

Table of contents

• Data （Download the complete data (1.5G) here.）

  • CLEF TAR 2017
  • CLEF TAR 2018
  • CLEF TAR 2019
  • TREC total recall track 2015/2016
  • TREC legal track 2010

• Code

  • baseline reproduction （Download the complete data (5.5G) for baseline reproduction here.）

    • AutoTar
    • Knee
    • Target
    • SCAL
    • Score distribution

  • autostop

    • tar_framework
    • tar_model

Data

• CLEF TAR 2017

It contains 42 topics selected from the complete topics by the organisor (see CLEF 2017 Technologically Assisted Reviews in Empirical Medicine Overview)

• CLEF TAR 2018

It contains 30 new topics released in CLEF TAR 2018 (see CLEF 2018 Technologically Assisted Reviews in Empirical Medicine Overview)

• CLEF TAR 2019

It contains 31 new topics released in CLEF TAR 2019 (see CLEF 2019 Technologically Assisted Reviews in Empirical Medicine Overview)

• TREC total recall track 2015/2016

It contains two subsets: athome1 contains 10 topics, and athome4 contains 34 topics. (See TREC 2016 Total Recall Track Overview)

• TREC legal track 2010

It contains 4 topics in the interactive task in the TREC 2010 Legal track. (See Overview of the TREC 2010 Legal Track).

Citation

@article{10.1145/3411755,
author = {Li, Dan and Kanoulas, Evangelos},
title = {When to Stop Reviewing in Technology-Assisted Reviews: Sampling from an Adaptive Distribution to Estimate Residual Relevant Documents},
year = {2020},
issue_date = {October 2020},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {38},
number = {4},
issn = {1046-8188},
url = {https://doi.org/10.1145/3411755},
doi = {10.1145/3411755},
journal = {ACM Trans. Inf. Syst.},
month = sep,
articleno = {41},
numpages = {36},