DO Thi Duyen - LE Ta Dang Khoa
- Packages:
Scrapy for crawling: http://doc.scrapy.org/en/latest/
$pip install scrapy
Selenium and Webdriver for JavaScript actions (e.g. click 'Show more'): https://pypi.org/project/selenium/
$pip install -U selenium
Download webdriver (chromedriver.exe/geckodriver.exe/...) and put it in
google_scholar/libs
For linux server:$sudo apt-get install -y chromium-browser
- Spiders:
google_scholar/spiders
- Get list of papers given user URLs
google_scholar/spiders/papers_spider.py
- Get list of papers which cited the given paper
google_scholar/spiders/citations_spider.py
- Run:
$python3 google_scholar/runner.py