SimulEval is a general evaluation framework for simultaneous translation on text and speech.
- python >= 3.7.0
git clone git@github.com:fairinternal/SimulEval.git
cd SimulEval
pip install -e .
Following is the evaluation of a dummy agent which operates wait-k (k = 3) policy and generates random words until the length of the generated words is the same as the number of all the source words. A tutorial can be found here.
cd examples
simuleval \
--agent dummy/dummy_waitk_text_agent.py \
--source data/src.txt \
--target data/tgt.txtSimulEval is licensed under Creative Commons BY-SA 4.0.
Please cite as:
@inproceedings{simuleval2020,
title = {Simuleval: An evaluation toolkit for simultaneous translation},
author = {Xutai Ma, Mohammad Javad Dousti, Changhan Wang, Jiatao Gu, Juan Pino},
booktitle = {Proceedings of the EMNLP},
year = {2020},
}