- Implement a web crawler
- japanese morphological analysis using janome
- Implement search engine
- Store in MongoDB
- Web frontend using Flask
More details are avairable from My Tech Blog(Japanese).
- Python 3.5
-
Clone repository
$ git clone git@github.com:maztak/pysearch.git
-
Install python packages
$ cd pysearch $ pip3 install -r requirements.txt -c constraints.txt
-
MongoDB settings
Install MongoDB
$
Install pymongo
$
Start mongo shell and create table(e.g. index table)
$ mongo > use index
in
config.py
, set the mongo url, like below.MONGO_URL = 'mongodb://127.0.0.1:27017/index'
If you want to use MongoDB with GUI, I recommend free software Robo 3T. Only Robo 3T is sufficient(No Studio 3T is needed).
-
Run
$ python3 manage.py crawler # build a index $ python3 manage.py webpage # access to http://127.0.0.1:9000