kanjirz50 / vnlp-outline

Describe Vietnamese NLP tools, resources and techniques related on Vietnamese.

Vietnamese Natural Language Processing(Xử lý ngôn ngữ tiếng Việt)

Outline

This page describes Vietnamese NLP tools, resources and some computation techniques related on Vietnamese. If there is something missing or wrong information, please create an issue or pull request.

(Sorry, I don't speak Vietnamese.)

Tools

Diacritics restoration

Vietnamese diacritics restoration tool

Github, demonstration

This tool is based on a point-wise diacritics restoration.

Word segmentation

DongDu

Github, website

DongDu is a fastest and high accuracy word segmentation tool based on SVMs.

vnTokenizer

website

vnTokenizer provides Java interface. I found some unofficial github repositories.

Part-of-Speech tagging

vnTagger

website

This tool provides Java interface.

Joint word segmentation and Part-of-Speech tagging

Vietnamese joint word segmentation and Part-of-Speech tagging based on SVM

Github, demonstration

This tool provides Python interface.

Vietnamese joint word segmentation and Part-of-Speech tagging based on CRF

Github, demonstration

This tool provides Python interface.

Other Tools

Vitk -- a Vietnamese text processing toolkit

Github

This toolkit is for large data processing, includes word segmentation, part-of-sppech tagging, dependency parsing.

Resources

Treebank

Vietnamese treebank

website

Corpus

VNESEcopus.txt

website

650 thousand sentences.

VNTQcorpus(small).txt

website

300 thousand sentences.

VNTQcorpus(big).txt

website

1.75 million sentences.

Bilingual Corpus

English-Vietnamese bilingual corpus

website

Technique

Unicode normalization

NFC is a one of the unicode normalization form. It is suitable for Vietnamese text preprocessing.

Syllable normalization

Articles

• Tuan Anh Luu and Kazuhide Yamamoto. A Point-wise Approach for Vietnamese Diacritics Restoration. Proceedings of the International Conference on Asian Language Processing (IALP 2012), pp.189-192 (2012.11)