- Install ckiptagger
- Downloading model file (require 2GB disk space)
- Follow https://github.com/ckiplab/ckiptagger
- python>=3.6
- tensorflow>=1.13.1,<2 / tensorflow-gpu>=1.13.1,<2 (one of them)
- gdown (optional, for downloading model files from google drive)
- Install beautifulsoup4, scikit-learn
$ sh ./utils/setup.sh
$ python ./tests/ckiptaggerTest.py
$ sh ./wordTokenization/launcher.sh
output files :
./wordTokenization/description.txt
./wordTokenization/output.txt
$ sh ./TF-IDF/launcher.sh
output files :
./TF-IDF/output.txt