Daemon stops processing commands after a while
kevinmehall opened this issue · comments
After a day or so, rash daemon stops processing commands. The process is still running, but the files are piling up in ~/.config/rash/data/record/command. If I kill the process and open a new terminal to restart the daemon, the new daemon successfully processes the json files.
Trying to poke at it with GDB, it looks like one thread is blocked in watchrecord.py line 64, a while True: time.sleep(1), and the other thread is blocked in a Queue.get in watchdog/observers/api.py, line 417
Fedora 18, Python 2.7.3, zsh 5.0.2
Hi, thanks for the report and narrowing issue. I have the same problem occasionally in my old laptop typically after sleep. I suspect this is a problem in the watchdog module [1] as you pointed out. Why don't you report an issue there?
The thing we can do in RASH is to add a simple polling method. This provides not only a workaround for this problem but also works with Python 3 and in some environments where file system notification is useless such as NFS.
At the moment, the easiest workaround I can think of is to call something like this from cron job to restart daemon periodically. Yea, I know it's not super cool...
rash init --daemon-opt=--restart > /dev/null
45ae53a adds --use-polling option. When specified, RASH uses polling instead of inotify. It is not perfect solution but it is all I can (want) do on RASH side. It is not clear if there is a problem in the watchdog inotify backend or somewhere else. If the former is the case, it should fix your problem.