Giters

kermitt2 / entity-fishing

A machine learning tool for fishing entities

Home Page:http://nerd.readthedocs.io/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Training on Arabic language

lecidhugo opened this issue · comments

commented

Hello,
Is there any document or guide on how to train on Arabic ?
Is this possible ? if yes what are the requirements ?

Thanks in advance,

commented

Hello @lecidhugo !

You can create the resources for a new language with https://github.com/kermitt2/grisp
The readme describes the process. It's an Hadoop process that is going to take a few hours.

Once done, you can start an environment for Arabic with entity-fishing, the knowledge base will be automatically build. Then you need to train a ranker and a selector model as described here -> https://nerd.readthedocs.io/en/latest/train.html#training-with-wikipedia

Loading the markupFull is the DB that is time consuming, it stores all the article text content.

You don't need to create embeddings if I remember well, it should work without them. However it improves a bit the disambiguation. This is also quite time consuming (it should be half day for Arabic given the number of articles).

There are 1,080,907 articles in Arabic, so it's a pretty big number, it should be doable and provide decent results.

commented

Thank you very much for your kind reply!
I will try to do it

commented

Note that Arabic is now supported by default, with already trained models and KB resources available, see the documentation.