This is a Scrapy-based crawler. A tutorial is at this url. The scraped information includes:
Conference, matched keywords, title, citation count, code url, pdf url, authors, abstract
The crawler scrapes accepted papers from top AI conferences, including:
- CVPR and ICCV since 2013.
- ECCV since 2018.
- AAAI since 1980.
- IJCAI since 2017.
- NIPS since 1987.
- ICML since 2017.
- ICLR 2018, 2019, 2021, and 2022.
pdf_url
may not work for ICML and ICLR (2022) as the website does not provide the url.
- ACM MM since 2001.
pdf url
is not available, as the ACM is not open-access.- The
download_delay
is set to 3s to not being banned.
- 28-OCT-2022
- Added a feature in which the target conferences can be specified in
main.py
. See Example 4.
- Added a feature in which the target conferences can be specified in
- 27-OCT-2022
- Added the crawler for ACM Multimedia.
- 20-OCT-2022
- Fixed a bug that falsely locates the paper pdf url for NIPS.
- 7-OCT-2022
- Rewrote
main.py
so that the crawler can run over all the conferences!
- Rewrote
- 6-OCT-2022
- Removed the use of
PorterStemmer()
fromnltk
as it involves false negative when querying.
- Removed the use of
pip install scrapy semanticscholar fuzzywuzzy git+https://github.com/sucv/paperCrawler.git
It's a Scrapy project. Simply cd to PaperCrawler/crawlconf/
then call the spider. Some examples are provided below.
scrapy crawl [conference name] -a years=[year1,year2,...,yearn] -a keys=[key1,key2,...,keyn] -a cc=[1 or 0] -o [output_filename.csv] -s JOBDIR=[checkpoint_folder]
conference name
: cvpr, iccv, eccv, aaai, ijcai, nips, icml, iclr, mm. Must be lowercase.year
: Four-digit numbers, use comma to separate.keys
: The abstract must contain at least one of the keywords. Use comma to separate.cc
: Set to 1 to count the citations using SemanticScholar API.output_filename
: The output csv filename. The outputs will attach to previous file if two commands share the same output filename.checkpoint_folder
: The folder to store the spider state.
scrapy crawl iccv -a years=2021,2019,2017 -a keys=video,emotion -o output.csv
The command above can scrape the information of all papers from ICCV2017, ICCV2019, ICCV2021, with "video" OR "emotion"
appeared in the abstracts. The results will be saved in output.csv
. It won't count the citations and save the
checkpoint. It scrapes really fast.
scrapy crawl nips -a years=2020,2021,2022 -a keys=video,emotion -a cc=1 -o output.csv
The command above works in the same manner, it also scrapes the number of citation for each paper. Note that the latter
is done using the free SemanticScholar API. Currently, I use the paper
title to query. Note that the API has a maximum request limit per second. Use this command with cautious, because it
could be dramatically time-consuming. (Or skip the citation count and post-process the output.csv
on your own.)
scrapy crawl ijcai -a years=2021,2020 -a keys=video -a cc=1 -o output.csv -s JOBDIR=folder1
The command above will save the scraping checkpoint in
a folder named folder1
. If the scraping process is interrupted by CTRL+C
or other incidents, simply execute the same
command so that the scraping can continue.
python main.py -confs [conf1,conf2,...,confn] -years [year1,year2,...,yearn] -keys [key1,key2,...,keyn] -cc 1
python main -confs cvpr,iccv,eccv -years 2018,2019,2020,2021,2022 -keys emotion,multimodal,multi-modal -cc 1
The command would scrape the papers, whose abstracts contain at least one keys, from CVPR, ICCV, and ECCV since 2018. In this case, the scraped data will be saved in data.csv
, which is defined in settings.py
. Note, specifycc
to 0
and exclude mm
from the confs
can greatly speed-up the process.