TerrorCat http://terra.cl.uzh.ch/terrorcat.html TODO POS-SETS * WHAT IS IT TerrorCat performs automatic ranking of translations -- i.e. given several translations (e.g. done by several MT systems) of the same source text, it will rank those translations according to their quality. * WHAT IS IT NOT TerrorCat is not an MT metric (like BLEU, TER or METEOR) -- you cannot use its output as an objective and independent estimate of translation quality; it can only rank translations, relative to other translations. * HOW DOES IT DO IT TerrorCat is based on automatic translation error categorization; it uses Addicter and Hjerson to automatically analyze and categorize translation errors of every hypothesis translation, then calculates the frequency of every error type per part-of-speech (in other words, frequencies of missing nouns/missing verbs/superfluous nouns/superfluous adjectives/misinflected nouns/misplaced nouns/etc. for every type of error and every part of speech), then uses those frequencies for pairwise comparison of the translation sentences (using a trained binary classifier), and finally computes the final ranking based on the outcome of all the pairwise comparisons. * HOW DO I APPLY IT First, take a look at the dependencies section down below. To rank a set of hypothesis translations, you need to 1. lemmatize and PoS-tag the source text, the reference and the hypothesis translations - save them all in the factored format "surface-form|pos-tag|lemma", for example ====== the|DT|the cars|NN|car were|VB|be small|JJ|small .|SENT|. ====== - name the source and reference files in the convention <test-set>.<language>.fact, for example newstest2011.fr.fact and newstest2011.en.fact - name the hypothesis files in the convention <test-set>.<language-pair>.<MT-system-ID>.fact, for example newstest2011.fr-en.uedin.fact, newstest2011.fr-en.cmu.fact and newstest2011.fr-en.cu-combo.fact - place them all in a single folder 2. launch TerrorCat's ranking with the script terrorcat.pl: ./terrorcat.pl [options] work-dir source-dir model-file > output.txt - work-dir is a path for saving auxilliary files, such as automatic error analysis output and such; you can specify a non-existent path and it will be created. To avoid having to re-generate the error analysis, always specify the same path -- as long as file names do not intersect, you can re-use the folder over several sessions (with different languages, etc.) - source-dir is the path to where you put the source texts and reference and hypothesis translations - model-file is the path to the pre-trained model for the pairwise classifier; you can download models from the homepage, or train your own - output.txt is the file where you want the ranking saved; the format is the same as for the WMT metric shared tasks (e.g. http://www.statmt.org/wmt12/metrics-task.html) - the options include -s to switch to sentence-level ranking (default: document-level ranking) -m to set the number of threads to use (default: 2) * HOW DO I TRAIN A NEW MODEL First, take a look at the dependencies section down below. To train a new model for TerrorCat you need a set of manually ranked translations -- the source text, reference and two or more hypothesis translations and their quality comparison on a per-sentence basis. 1. launch the training with the script train-model.pl: ./train-model.pl [options] work-dir man-rank-file source-dir model-file - work-dir is a path for saving auxilliary files, such as automatic error analysis output and such; you can specify a non-existent path and it will be created. To avoid having to re-generate the error analysis, always specify the same path -- as long as file names do not intersect, you can re-use the folder over several sessions (with different languages, etc.) - source-dir is where you placed the source text, reference and hypothesis translation files - man-rank-file is the file with the manual translation rankings on a per-sentence basis; the format is ====== language-pair,test-set-id,line-nr,sys-1-id,sys-1-rank,sys-2-id,sys-2-rank,... ... ====== for example ====== fr-en,newstest2011,42,uedin,1,cmu,2,cu-combo,1 ====== and a couple of hundred (or thousand) more of such lines. - model-file is the path to where to save the model file (to pass to the applying script terrorcat.pl afterwords) - the options include -m to set the number of threads to use (default: 2) * HOW DO I META-EVALUATE If you have a trained model and a manual ranking file for data, that was not involved in training the model, you can calculate the correlation between TerrorCat's ranking and manual ranking. To do so, use the script metaeval.sh: ./metaeval.sh man-ranks auto-ranks - man-ranks is the file with manual ranking data in the same format as for training models (see above) - auto-ranks is the output of TerrorCat's ranking script; metaeval.sh will automatically guess, whether the ranking output was on document or sentence level and calculate the appropriate correlation coefficient (Spearman's Rho for document level and Kendall's Tau for sentence level) * WHAT ARE THE DEPENDENCIES TerrorCat uses Addicter, Hjerson and Weka: - https://wiki.ufal.ms.mff.cuni.cz/user:zeman:addicter - http://www.dfki.de/~mapo02/hjerson/ - http://www.cs.waikato.ac.nz/ml/weka/ Weka's version -- not sure, but with 3.6.6 (and probably newer ones) it works. The paths to these Software packages can be set it the "config.ini" file. Also, TerrorCat has only been tested on Linux, but in theory it could work elsewhere (e.g. on Windows via Cygwin, or on MacOS). It contains Perl scripts, Bash scripts and GNU Make scripts, so Perl, Bash and Make are also a requirement. * HOW WELL DOES IT WORK The correlation with human judgements varies, depending on the language, text domain, size of training data used to create the model, etc. When evaluated on WMT'2011 data, document-level correlations were: English: 0.854 French: 0.919 German: 0.879 Spanish: 0.943 Czech: 0.915 and sentence-level correlations were: English: 0.305 French: 0.3 German: 0.175 Spanish: 0.254 Czech: 0.208 See the homepage for more recent evaluation results.