tinok / techreview_arxiv_scrape

techreview_arxiv_scrape

The data and partial code behind the story "We analyzed 16,625 papers to figure out where AI is headed next" on MIT Technology Review.

Files

• arxivscraper.py: A modified version of Mahdisadjadi/arxivscraper
• arxivscraper_all_cs_ai.ipynb: The code for scraping metadata from the arXiv
• cs_ai_through_2018-11-18.csv: The metadata for all scientific papers published to the AI section of the arXiv through November 18, 2018
• arxivscraper_ai_only_analysis.ipynb: The code for counting all of the words and ngrams by year, and for calculating those with the greatest losses and gains

Data columns

• id: arXiv paper id
• title: paper title
• categories: the arXiv categories under which the paper was filed
• abstract: paper abstract
• doi: digitial object identifier
• created: date of paper's upload
• updated: date of paper's latest update
• authors: paper authors last names
• year: year and month of paper's upload

Source

• arXiv.org > Artificial intelligence