Giters

rohit-dua / BUB

BUB : Book Uploader Bot

Home Page:http://tools.wmflabs.org/bub/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Set up bigbrother restarts

nemobis opened this issue · comments

commented

As you are already using jstart, are you ready to use bigbrother? https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid#Bigbrother
That should help getting all the workers continuously running without human supervision and with known-working parameters.

commented

For now I added:

jstart -l release=trusty -N worker0 -o /data/project/bub/public_html/BUB/bot/worker.log -e /data/project/bub/public_html/BUB/bot/worker.log -wd /data/project/bub/public_html/BUB/bot/ -mem 1000m python worker.py
jstart -l release=trusty -N uc -o /data/project/bub/public_html/BUB/bot/upload_checker.log -e /data/project/bub/public_html/BUB/bot/upload_checker.log -wd /data/project/bub/public_html/BUB/bot/ python upload_checker.py
jstart -l release=trusty -N mworker3-1 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -wd /data/project/bub/public_html/BUB/bot/ -mem 1000m python mass_worker.py 3
jstart -l release=trusty -N mworker3-2 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -wd /data/project/bub/public_html/BUB/bot/ -mem 1000m python mass_worker.py 3
jstart -l release=trusty -N mworker3-3 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -wd /data/project/bub/public_html/BUB/bot/ -mem 1000m python mass_worker.py 3

Except that it doesn't work, although the jstart command works on the command line...

2016-02-08 21:07:21 error: /data/project/bub/.bigbrotherrc:1: command not supported
2016-02-08 21:07:23 error: /data/project/bub/.bigbrotherrc:2: command not supported
2016-02-08 21:07:24 error: /data/project/bub/.bigbrotherrc:3: command not supported
2016-02-08 21:07:25 error: /data/project/bub/.bigbrotherrc:4: command not supported
2016-02-08 21:07:27 error: /data/project/bub/.bigbrotherrc:5: command not supported
commented

I could not get bigbrother to work, but a cronjob with a script to restart the bot is doing well so far:

tools.bub@tools-bastion-05:~$ crontab -l | tail -1
7 */6 * * * /usr/bin/jsub -N cron-tools.bub-2 -once -quiet /data/project/bub/bots_restart.sh
tools.bub@tools-bastion-05:~$ cat bots_restart.sh
#!/bin/bash
cd /data/project/bub/public_html/BUB/bot/
jsub -once -continuous -l release=trusty -N mworker3-2 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -cwd -mem 1000m ./mass_worker.py 3
jsub -once -continuous -l release=trusty -N mworker3-1 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -cwd -mem 1000m ./mass_worker.py 3
jsub -once -continuous -l release=trusty -N mworker3-3 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -cwd -mem 1000m ./mass_worker.py 3
jsub -once -continuous -l release=trusty -N mworker2-1 -o /data/project/bub/public_html/BUB/bot/mass_worker.log -e /data/project/bub/public_html/BUB/bot/mass_worker.log -cwd -mem 1000m ./mass_worker.py 2
commented

There is no jsub either on the exec hosts and tools.bub can't ssh into the bastion host, so I just set up a tmux screen with an infinite loop running the restarts...