This repo is a collection of scripts to load and transform article and talk data from Wikipedia dumps, as well as user data from Wikipedia API.
Requirements: python 3.5, spark 2.2, docker, docker-compose
Install python packages and pbzip2:
pip install luigi pyspark gensim
sudo apt-get install pbzip2
Start up the mongodb container:
docker-compose up
Run via a local luigi scheduler:
PYTHONPATH='.' luigi --module wiki_dump WikiDump --local-scheduler --workers=4
Edit wiki language and mongo collection name in wiki_users.py:
language = 'en'
users_collection = con['wikipedia']['en_users']
Run the script:
python wiki_users.py
A luigi pipeline is set up to download wikipedia XML page dumps and language link SQL dumps. They are transformed and combined using PySpark and then loaded into MongoDB.
Fields of the created collection wikipedia.en_articles (wikipedia.ru_articles):
- id (int) // wikipedia page ID
- title (str)
- text (str) // text in wiki markup
- sections (str[]) // cleaned article sections
- timestamp (datetime)
- redirects (str[]) // titles of the pages that redirect to the article
- translation_id (int) // ID of the page in another language
wiki_users.py is set up to run continuously, fetching users from the wikipedia API and updating user records in MongoDB. If the script is stopped abruptly, it will continue from the last updated user in the database on the next run.
API docs: https://www.mediawiki.org/wiki/API:Allusers
Fields:
- userid
- name
- blockid
- blockedby
- blockedbyid
- blockedtimestamp
- blockreason
- blockexpiry
- editcount
- registration
- groups
- updated // added field to track progress