HanleiZhang / THUMT

An open-source neural machine translation toolkit developed by Tsinghua Natural Language Processing Group

THUMT: An Open Source Toolkit for Neural Machine Translation

Contents

• Introduction
• Online Demo
• Implementations
• Notable Features
• License
• Citation
• Development Team
• Contributors
• Contact
• Derivative Repositories

Introduction

Machine translation is a natural language processing task that aims to translate natural languages using computers automatically. Recent several years have witnessed the rapid development of end-to-end neural machine translation, which has become the new mainstream method in practical MT systems.

THUMT is an open-source toolkit for neural machine translation developed by the Natural Language Processing Group at Tsinghua University. The website of THUMT is: http://thumt.thunlp.org/.

Online Demo

The online demo of THUMT is available at http://translate.thumt.cn/. The languages involved include Ancient Chinese, Arabic, Chinese, English, French, German, Indonesian, Japanese, Portugese, Russian, and Spanish.

Implementations

THUMT has currently two main implementations:

• THUMT-TensorFlow: a new implementation developed with TensorFlow. It implements the sequence-to-sequence model (Seq2Seq) (Sutskever et al., 2014), the standard attention-based model (RNNsearch) (Bahdanau et al., 2014), and the Transformer model (Transformer) (Vaswani et al., 2017).

• THUMT-Theano: the original project developed with Theano, which is no longer updated because MLA put an end to Theano. It implements the standard attention-based model (RNNsearch) (Bahdanau et al., 2014), minimum risk training (MRT) (Shen et al., 2016) for optimizing model parameters with respect to evaluation metrics, semi-supervised training (SST) (Cheng et al., 2016) for exploiting monolingual corpora to learn bi-directional translation models, and layer-wise relevance propagation (LRP) (Ding et al., 2017) for visualizing and anlayzing RNNsearch.

The following table summarizes the features of two implementations:

Implementation	Model	Criterion	Optimizer	LRP
Theano	RNNsearch	MLE, MRT, SST	SGD, AdaDelta, Adam	RNNsearch
TensorFlow	Seq2Seq, RNNsearch, Transformer	MLE	Adam	RNNsearch, Transformer

We recommend using THUMT-TensorFlow, which delivers better translation performance than THUMT-Theano. We will keep adding new features to THUMT-TensorFlow.

It is also possible to exploit layer-wise relevance propagation to visualize the relevance between source and target words with THUMT:

Notable Features

• Transformer (Vaswani et al., 2017)
• Multi-GPU training & decoding
• Distributed training
• Float16 training
• Model ensemble & Averaging
• Relative position embedding (Shaw et al., 2018)
• Visualization with layer-wise relevance propagation (LRP) (Ding et al., 2017)

License

The source code is dual licensed. Open source licensing is under the BSD-3-Clause, which allows free use for research purposes. For commercial licensing, please email [e-mail].

Citation

Please cite the following paper:

Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng, Maosong Sun, Huanbo Luan, Yang Liu. 2017. THUMT: An Open Source Toolkit for Neural Machine Translation. arXiv:1706.06415.

Development Team

Project leaders: Maosong Sun, Yang Liu, Huanbo Luan

Project members: Jiacheng Zhang, Yanzhuo Ding, Shiqi Shen, Yong Cheng

Contributors

• Zhixing Tan (Xiamen University)

Contact

If you have questions, suggestions and bug reports, please email [e-mail].

Derivative Repositories

• Document-Transformer (Improving the Transformer Translation Model with Document-Level Context)
• PR4NMT (Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization)