penzant / mctest_annotation

Annotations of MCTest (and SQuAD) with "prerequisite skills" in reading comprehension

This repository contains

• Links for annotations of MCTest and SQuAD that are reported in Sugawara and Aizawa (2016) and Sugawara et al. (2017)
• Per-question(option) results of two RC systems in MCTest (dev)

Anntation of the MC160/MC500 development sets of MCTest

• https://docs.google.com/spreadsheets/d/1yHt8IcPz2Nxrc_Mk_wMIxH3uL7pFHj_jLee4OWD12Sw/edit?usp=sharing

Per-option results of RC systems in the MC160/MC500 development sets of MCtest

• Baseline and Yin ABCNN: mc{160,500}.dev_{baseline,habcnn}.results in this repository
• Smith et al. (2015) RTE/NoRTE systems: https://github.com/elleryjsmith/UCLMCTest

Annotation of the development set of SQuAD (v1.1)

• https://docs.google.com/spreadsheets/d/1sSJBhC9AiokAJ4nx_O7J1MUh2SuA3gbOxX-lQ__SQX4/edit?usp=sharing
  • in Sugawara and Aizawa (2016)

References

• MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text
  • Richardson et al. (2013)
  • http://aclweb.org/anthology/D13-1020
  • https://www.microsoft.com/en-us/research/publication/mctest-challenge-dataset-open-domain-machine-comprehension-text/
• A Strong Lexical Matching Method for the Machine Comprehension Test
  • Smith et al. (2015)
  • https://aclweb.org/anthology/D/D15/D15-1197.pdf
  • https://github.com/elleryjsmith/UCLMCTest
• Attention-Based Convolutional Neural Network for Machine Comprehension
  • Yin et al. (2016)
  • https://arxiv.org/abs/1602.04341
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
  • Rajpurkar et al. (2016)
  • https://arxiv.org/abs/1606.05250
  • https://stanford-qa.com/
• An Analysis of Prerequisite Skills for Reading Comprehension
  • Sugawara and Aizawa (2016)
  • http://aclweb.org/anthology/W/W16/W16-6001.pdf
• Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems
  • Sugawara et al. (2017)
  • http://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14614/14082