digitalepidemiologylab / mkondo

A library for downloading and managing Twitter streams of data.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Mkondo is Kiswahili for 'stream'. 

This library contains code that helps download Twitter Streaming API data and manage that data. 

It relies on the Salathegroup's fork of Tweepy
(https://github.com/salathegroup/tweepy) which allows the raw JSON received to
be written out to disk. 

Mkondo provides the following features: 

1. extends Tweepy classes to include appropriate backoffs for Twitter's Streaming API. 

2. code to store and backup tweets from the Streaming API (this includes
rotating directories and data files when appropriate and logging errors as
necessary)

3. classes to help parse and analyze the tweets as stored on disk.

4. a class to help use the Twitter REST API continuously, taking appropriate
breaks when the hourly limit for REST API calls is reached. 

We use this module to continuously collect data from Twitter with limited downtime. 

khandelwal@psu.edu, salathe@psu.edu

About

A library for downloading and managing Twitter streams of data.


Languages

Language:Python 100.0%