Damianus04 / id-nlp-resource

A list of Indonesian NLP resources.

Indonesian NLP resources

Language modeling

1. Kompas online collection. This corpus contains Kompas online news articles from 2001-2002. See here for more info and citations.
2. Tempo online collection. This corpus contains Tempo online news articles from 2000-2002. See here for more info and citations.
3. OSCAR. This large corpus contains articles from many sources crawled by CommonCrawl and extracted by ALMAnaCH. In total there are 4B words tokens and 2B word types. (NOTE: Contains strong language, mostly coming from gambling sites.)

POS tagging

1. PANL10N POS tagging. This corpus has 39K sentences and 900K word tokens.
2. IDN tagged corpus. This corpus contains 10K sentences and 250K word tokens. The POS tags are annotated manually.

Sentiment analysis

1. Aspect and Opinion Terms Extraction for Hotel Reviews. The corpus consists of 5000 hotel reviews from Airy (78K tokens) with 5 labels. The paper is available on arXiv.
2. Aspect-Based Sentiment Analysis. A text classification resource for multi-label aspect categorization.

Syntactic parsing

1. Indonesian Treebank. This corpus contains 1K parsed sentences. (constituency parsing)
2. UD Indonesian. This corpus is provided by Universal Dependencies. Training, development, and testing split are already provided. (dependency parsing)

Machine translation

1. PANL10N EN-ID news parallel corpus. This corpus has sentences from news articles from several categories: economy (6K sentences), international (6K sentences), science (6K sentences), and sport (4K sentences).
2. PANL10N Indonesian translation of Penn treebank. This corpus contains Indonesian translation of the Penn treebank. In total there are 24K sentences.
3. OPUS (Open Parallel Corpus). This site contains parallel corpora of Indonesian and other languages based on openly available resources (e.g., OpenSubtitles).
4. IDENTICv1.0 [paper]. Indonesian (ID)-English (EN). 45k sentences/~1M tokens (ID). Domain: science, sport, international, economy, news article, movie subtitle. It may overlap with PANL10N corpus. The dataset has versions with raw and tokenized sentences, and in CoNLL format.
5. IWSLT2017 [paper]. ID-EN. ~100k sentences. TEDtalk subtitles (spoken language). NOTE: the test set tst2017-plus provided contains a small part of the train data (as mentioned here).
6. Asian Language Treebank [paper]. ID, EN, and some Asian languages (mostly South East Asian). 20k sentences. Domain: News.

Word normalization

1. Colloquial Indonesian Lexicon. This lexicon consists of 3592 unique colloquial tokens that are mapped onto 1742 unique lemmas. The full description of this lexicon can be seen in the paper.

Text summarization

1. IndoSum. A collection of 20K online news article-summary pairs belonging to 6 categories and 10 sources. It has both abstractive summaries and extractive labels.

Text classification

1. SMS Spam. This corpus contains 1143 sentences that have been labeled with normal message, fraud, promotion. It is provided by http://nlp.yuliadi.pro/dataset
2. Hate Speech Detection. This dataset consists of 713 tweets in the Indonesian language with 453 non hate speech and 260 hate speech tweets.
3. Abusive Language Detection. A collection of tweets for abusive language detection in Indonesian social media. It consists of two types of labelling, abusive/not abusive and not abusive/abusive but not offensive/offensive. It also has its own colloquial Indonesian lexicon.

Speech recognition

1. TITML-IDN speech corpus. The corpus contains 20 speakers (11 male and 9 female), where each of the speaker speaks 343 utterances. The utterances are phonetically balanced. The corpus itself is free to use for academic/non-commercial usage, but interested party should make a formal request via email to the institution. The procedure is listed here.
2. Indonesian Speech Recognition. A small corpus of 50 utterances by a single male speaker. Disclaimer: This is a school project, do not use it for any important tasks. The author is not responsible for the undesired results of using the data provided here.
3. CMU Wilderness Multilingual Speech Dataset. A dataset of over 700 different languages providing audio, aligned texts, and word pronunciations. One of the languages is Indonesian. The utterances are read from the bible, which is recorded by bible.is.