kethort / TwitterLDATopicModeling

Uses topic modeling to identify context between follower relationships of Twitter users

Description

Twitter users often associate and socialize with other users based on similar interests. The Tweets of these users can be classified using a trained LDA model to automate the discovery of their similarities.

Prerequisites

Python 2.7 is recommended since the pattern library is currently incompatible with most Python 3 versions.

Python 3.6 can be used with the pattern library, though it may need to be built from source since most newer Linux distributions don't come with it pre-installed. The commands to build Python 3.6 from source are provided in the linux_setup_py3.6.sh script.

Installing

Linux

Download:

git clone https://github.com/kethort/twitter_LDA_topic_modeling.git

Run bash script:

./linux_setup_py3.6.sh

Python pip requirements included in these files:

# for Python 2.7
pip install -r requirements_py2.txt

# for Python 3
pip install -r requirements_py3.txt

Link to the simple-wikipedia dump:

https://dumps.wikimedia.org/simplewiki/latest/simplewiki-latest-pages-articles.xml.bz2

Mac osx

The installation is very similar to the linux installation:

extra install instructions in osx_setup_py3.6.info

pip install -r requirements_py3_OSX.txt

Process

1. Get user and follower ids by location - twitter_user_grabber.py
2. Download Tweets for each user - get_community_tweets.py
3. Create an LDA model from a corpus of documents - create_LDA_model.py
4. Generate topic probability distributions for Tweet documents - tweets_on_LDA.py
5. Calculate distances between Tweet documents and graph them - plot_distances.py

Sample Visualizations

Built With

• Gensim - Package for creating LDA model
• pyLDAvis - Package for visualizing LDA model
• Tweepy - Package for interacting with Twitter REST API
• NLTK - Package for stopword management and tokenization