research-gate-crawler

Crawling ResearchGate papers' Title & Abstract info and written into a CSV file.

Introduction

In default, the crawler will retrieve all articles with below strategy:

Only Article, not Conference Paper, Project, or any else
The Article must contain one of the term in the string: drone uav remote-sensing "remote sensing"
The Article must have Abstract
The Article must have Announced date from year 2000

Modifying crawling parameters

You may visit https://www.researchgate.net/search.Search.html?type=publication&query=drone%20uav%20remote-sensing to figure out all possible params

# crawler.py
base_url = 'https://www.researchgate.net/search.SearchBox.loadMore.html'

# Keywords used to search papers
# Looks like the query operator is OR,
# eg., searching 'drone' returns totalItems=7467, whereas, searching 'drone uav' returns totalItems=28388
terms = 'drone uav remote-sensing "remote sensing"'

# Searching Publications, not Authors or Reviews...
search_type = 'publication'
# Type of publication we need to search is Article, not Conference Paper, Posters, or Others...
publication_type = 'article'

# From experiments, the max items we can crawl for each request is 300
limit = 300

Notice:

From our tests, limitation for crawling is 300 items in each request.
Besides, there is Request time-rate limitation. By adding cookies, we can bypass this kind of error.

Launch the crawler

$ python crawler.py
Fetching 300/178863 (0%)
https://www.researchgate.net/search.SearchBox.loadMore.html?query=drone+uav+remote-sensing+%22remote+sensing%22&type=publication&subfilter%5BpublicationType%5D=article&offset=300&limit=300
retrieved (192) & ignored (108)
...

Results

csv files, that support UTF-8 encoding, named ResearchGate-YYYY-YYYY-MM-DD.csv will be created in the same location of the crawlers/caller

In which,

The first YYYY is the announced year of publications.
YYYY-MM-DD is the crawling date.

Ex.,

ResearchGate-2007-2020-01-06.csv
ResearchGate-2008-2020-01-06.csv
...

Data Collection (How-to)

The crawled data was taken on Jan 10, 2020 (2020-Jan-10.zip)
The results are 21 CSV files for keywords drone uav remote-sensing "remote sensing". Each CSV is for one year, from 2000 to 2020.
There are 178968 articles in total
There are 131180 (73.3%) articles collected and placed into their appropriate CSV
And, 47788 (26.7%) articles were ignored because of un-matched criteria (from year 2000 and must have the abstract)

Technology

The Crawler uses requests library to send a GET query to get the needed information returned in JSON format.

Future work

Figure out to extract Keywords from Title & Abstract

I am welcome any effort to enhance this open-source project.

Thank you.

lhviet / research-gate-crawler