DFKI-NLP / tacrev

[ACL 20] TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task

TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [Paper]

Table of Contents

• Overview
• Requirements
• Installation
• Patch TACRED
• Experiments
• Citation
• License

🔭  Overview

Path	Description
dataset/	The experiment notebooks expect the patched TACRED dataset splits to be stored here.
notebooks/	This directory contains the notebooks that we used to produce the results in the paper.
patch/	This directory contains the patches for dev and test split of the original TACRED.
results/	This directory contains the predictions of all models on dev (dev_results/) and test split (test_results/).
scripts/	This directory contains scripts, e.g., to apply the patch to TACRED dev or test split.

✅  Requirements

The code is tested with:

• Python 3.7
• Mysql Config 5.7 (required by Errudite)

  # Minimal dependency install (Ubuntu)
  sudo apt install default-libmysqlclient-dev
  

🚀  Installation

From source

git clone https://github.com/DFKI-NLP/tacrev
cd tacrev
pip install -r requirements.txt  # only necessary for notebooks

💡  Patch the original TACRED

Dev Split

python scripts/apply_tacred_patch.py \
  --dataset-file <TACRED DIR>/dev.json \
  --patch-file ./patch/dev_patch.json \
  --output-file ./dataset/dev_rev.json

md5 checksum of patched dev split: ce23ba10ca15bde94a3f733679bf1b05

Test Split

python scripts/apply_tacred_patch.py \
  --dataset-file <TACRED DIR>/test.json \
  --patch-file ./patch/test_patch.json \
  --output-file ./dataset/test_rev.json

md5 checksum of patched test split: dbcce82f5ab67fbfd1062db6cc6b66cd

🔬  Experiments

• notebooks/tables.ipynb produces the tables presented in the paper.
• notebooks/error_groups.ipynb produces the error group bar chart from the paper.

📚  Citation

If you find the code or dataset patch helpful, please cite the following paper:

@inproceedings{alt-etal-2020-tacrev,
    title={TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task},
    author={Christoph Alt and Aleksandra Gabryszak and Leonhard Hennig},
    year={2020},
    booktitle={Proceedings of ACL},
    url={https://arxiv.org/abs/2004.14855}
}

📘  License

The code is released under the under terms of the MIT License.