This repository contains supplementary material for the publication:
Antonio Toral, Sheila Castilho, Ke Hu and Andy Way. 2018. Attaining the Unattainable? Reassessing Claims of Human Parity in Neural Machine Translation. WMT.
-
export_appraise/ export of judgements from Appraise, and derived rankings, clusters and inter annotator agreement
- export_appraise.sh script that runs all the steps
- export/ judgements exported from Appraise in csv format for all the 49 documents
- clusters/ resulting clusters
- plot_cis.ods calculations and plot of Trueskill's confidence intervals
- wmt-trueskill/ third-party code to calculate rankings and clusters
- compute_agreement_scores.py third-party code to calculate IAA
-
import_appraise/
- xml/ 1 file in Appraise XML input format for each of the 169 documents from the WMT2017 data for the Chinese-English language pair. The order of the documents is randomised. Each document contains as metadata its original document identifier (e.g. sina0812.news.doc-ifxuxhas1768823) and the original language in which the document was written (en|zh)
- plain/ 5 plain text files per document: source (sl), human translation (tl), Microsoft's output (c6) and Google's output (gg)
-
regression/ R noteboook with a step-by-step regression analysis. We augment Microsoft's data with several additional predictors: sentence length, original language of the source text and document identifier.
-
ttr/ type-token ratio calculations