Parallelization of langid using joblib
-
Clone the langid repo:
git clone https://github.com/saffsd/langid.py.git
-
cd langid/langid.py/langid
-
Install requirements :
pip install -r requirements.txt
-
Create a directory
parallel_runs/
andn
directories inside this.n: number of parallel runs
. Generally, for a 48 core server, you can safely use around 40 threads. Simply do this with a bash loop.