######## Overview ######## Twap is a very hacky app that uses the tweetstream library and django to query the twitter streaming api for tweets filtered by terms in a searchlist. Primarily we used this to trag tweets based on hashtags related to occupy wall street so we can archive these for some kind of analysis later. Requirements ============ Additional libraries required to run the application are listed in requirements.txt. I recommend using Pip to install the requirements in a virtualenv. Setup ========== * Copy localsettings-dist.py to localsettings.py and configure as needed. * In localsettings.py add your TWITTER_USER and TWITTER_PASSWORD information. * Add a list of terms to filter tweets by in settings.SEARCHLIST * Syncdb and follow other normal Django setup processes. Note this does NOT need to be setup on apache. It mostly runs via manage.py Starting & Stopping Harvest =========================== ** Stopping Harvest ** I made twap to be hard to kill though it seems to lock up ever few days. I believe this is because the MySQL log fills up but I'm not sure. Anyway, if you need to restart the app you will have to *manually kill the app* by using the kill command. To do this you'll need the PID of the thread which is easily obtained by typing:: > ps aux | grep twitterharvest This will list all the jobs running with that name. I often run this as 'SCREEN' (see below) and here is an example of the output you will get:: sturnbu 2196 0.0 0.0 5248 528 ? Ss Oct25 0:07 SCREEN python manage.py twitterharvest sturnbu 2197 1.0 61.1 3096908 2438712 pts/1 Ss+ Oct25 29:04 python manage.py twitterharvest sturnbu 18738 0.0 0.0 4156 876 pts/3 S+ 12:41 0:00 grep --color=auto twitterharvest The PID is the number right after the user name and in this example I will want to kill PID '2196' but ** that value will differ in each case ** so use the command above to get the current PID. If in doubt just kill any PID with 'twitterharvest' in it but you should only have to kill the version with 'SCREEN' in it's process. To Kill a process you only need to type:: > kill <PID> In this example that would be:: > kill 2196 You can ensure the thread was killed by running the ps command again listed above but this time you should only see the line with 'grep' in it, which is the thread you created by running the ps command, and that is expected. ** Starting Harvest ** Harvest is run vi a manage.py command and if the application is installed and configured right you should only need to run the 'twitterharvest' command from the applicaiton root directory like so:: > screen python manage.py twitterharvest By using the screen command you should be able to close out the window and the harvest will continue to run even after you log out. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . _________ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ./ It’s a twap! \ . . . . . . . . . . . . . . . . _,,,--~~~~~~~~--,_ . . . .\ ._________/ . . . . . . . . . . . . . . ,-‘ : : : :::: :::: :: : : : : :º ‘-, . . \/. . . . . . . . . . . . . . . . . . . . . . .,-‘ :: : : :::: :::: :::: :::: : : :o : ‘-, . . . . . . . . . . . . . . . . . . . . . ,-‘ :: ::: :: : : :: :::: :::: :: : : : : :O ‘-, . . . . . . . . . . . . . . . . . . .,-‘ : :: :: :: :: :: : : : : : , : : :º :::: :::: ::’; . . . . . . . . . . . . . . . . .,-‘ / / : :: :: :: :: : : :::: :::-, ;; ;; ;; ;; ;; ;; ;\ . . . . . . . . . . . . . . . . /,-‘,’ :: : : : : : : : : :: :: :: : ‘-, ;; ;; ;; ;; ;; ;;| . . . . . . . . . . . . . . /,’,-‘ :: :: :: :: :: :: :: : ::_,-~~,_’-, ;; ;; ;; ;; | . . . . . . . . . . . . _/ :,’ :/ :: :: :: : : :: :: _,-‘/ : ,-‘;’-‘’’’’~-, ;; ;; ;;,’ . . . . . . . . . . . ,-‘ / : : : : : : ,-‘’’ : : :,--‘’ :|| /,-‘-‘--‘’’__,’’’ \ ;; ;,-‘ . . . . . . . . . . . \ :/,, : : : _,-‘ --,,_ : : \ :\ ||/ /,-‘-‘x### ::\ \ ;;/ . . . . . . . . . . . . . . \/ /---‘’’’ : \ #\ : :\ : : \ :\ \| | : (O##º : :/ /-‘’ . . . . . . . . . . . . . . . /,’____ : :\ ‘-#\ : \, : :\ :\ \ \ : ‘-,___,-‘,-`-,, . . . . . . . . . . . . . . . ‘ ) : : : :’’’’--,,--,,,,,,¯ \ \ :: ::--,,_’’-,,’’’¯ :’- :’-, . . . . . . . . . . . . . .) : : : : : : ,, : ‘’’’~~~~’ \ :: :: :: :’’’’’¯ :: ,-‘ :,/\ . . . . . . . . . . . . . .\,/ /|\\| | :/ / : : : : : : : ,’-, :: :: :: :: ::,--‘’ :,-‘ \ \ . . . . . . . . . . . . .\\’|\\ \|/ ‘/ / :: :_--,, : , | )’; :: :: :: :,-‘’ : ,-‘ : : :\ \, . . . . . . . . . . ./¯ :| \ |\ : |/\ :: ::----, :\/ :|/ :: :: ,-‘’ : :,-‘ : : : : : : ‘’-,,_ . . . . . . ..| : : :/ ‘’-(, :: :: :: ‘’’’’~,,,,,’’ :: ,-‘’ : :,-‘ : : : : : : : : :,-‘’’\\ . . . . . ,-‘ : : : | : : ‘’) : : :¯’’’’~-,: : ,--‘’’ : :,-‘’ : : : : : : : : : ,-‘ :¯’’’’’-,_ . ./ : : : : :’-, :: | :: :: :: _,,-‘’’’¯ : ,--‘’ : : : : : : : : : : : / : : : : : : :’’-, / : : : : : -, :¯’’’’’’’’’’’¯ : : _,,-~’’ : : : : : : : : : : : : : :| : : : : : : : : : : : : : : : :¯’’~~~~~~’’’ : : : : : : : : : : : : : : : : : : | : : : : : : : : :