Giters

Supervisor / initscripts

User-contributed OS init scripts for Supervisor

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

[Debian] Systemd loading configuration twice

OneCricketeer opened this issue · comments

commented

Quick background

  • Running a Debian-like distribution and installed supervisor via apt-get.
  • Using supervisor to run a Python Tornado Web Server on a total of 4 ports which is load-balanced via an Nginx reverse proxy

Problem

I am seeing 2 sets of my Python configuration for a total of 8 python processes, 2 for each of the 4 ports that I run my web-app on. 1 of these sets seems to randomly be started and not managed by supervisor. I have no idea where this set could be started from or how to properly manage it.

For example

Checking the supervisor service

sudo systemctl status supervisor

You can see there is 8 distinct processes, but 2 sets of port numbers (8000-8003).

● supervisor.service - LSB: Start/stop supervisor
   Loaded: loaded (/etc/init.d/supervisor)
   Active: active (running) since Wed 2015-12-23 18:19:58 CST; 13min ago
  Process: 754 ExecStart=/etc/init.d/supervisor start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/supervisor.service
           ├─760 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
           ├─765 python server.py --port=8002
           ├─770 python server.py --port=8003
           ├─771 python server.py --port=8000
           ├─772 python server.py --port=8001
           ├─777 python server.py --port=8001
           ├─778 python server.py --port=8002
           ├─779 python server.py --port=8003
           └─780 python server.py --port=8000
Can see the same thing with ps

sudo ps -aux | egrep "supervisor|python"

root       760  0.2  2.7  14496  9892 ?        Ss   18:19   0:01 /usr/bin/python /usr/bin/supervisord -c /etc/supervisor/supervisord.conf
www-data   765  0.3  3.9  16860 14156 ?        S    18:19   0:02 python server.py --port=8002
www-data   770  0.3  3.9  16872 14216 ?        S    18:19   0:02 python server.py --port=8003
www-data   771  0.3  3.8  16872 14008 ?        S    18:19   0:02 python server.py --port=8000
www-data   772  0.3  3.9  16860 14220 ?        S    18:19   0:02 python server.py --port=8001
www-data   777  0.0  3.4  17116 12420 ?        S    18:20   0:00 python server.py --port=8001
www-data   778  0.0  3.0  16860 11184 ?        S    18:20   0:00 python server.py --port=8002
www-data   779  0.0  3.1  16872 11264 ?        S    18:20   0:00 python server.py --port=8003
www-data   780  0.0  3.4  17128 12456 ?        S    18:20   0:00 python server.py --port=8000
Stop supervisor via systemctl
sudo systemctl stop supervisor
sudo systemctl status supervisor

● supervisor.service - LSB: Start/stop supervisor
   Loaded: loaded (/etc/init.d/supervisor)
   Active: inactive (dead)

sudo supervisorctl status
unix:///var/run/supervisor.sock no such file

So, supervisor has destroyed the socket file, BUT running ps again shows

sudo ps -aux | egrep "supervisor|python"

www-data   777  0.0  3.4  17116 12420 ?        S    18:20   0:00 python server.py --port=8001
www-data   778  0.0  3.0  16860 11184 ?        S    18:20   0:00 python server.py --port=8002
www-data   779  0.0  3.1  16872 11264 ?        S    18:20   0:00 python server.py --port=8003
www-data   780  0.0  3.4  17128 12456 ?        S    18:20   0:00 python server.py --port=8000
commented

Here are my relevant config files. And I am using the default initscript that was included with apt-get install supervisor. That is located at /etc/init.d/supervisor.

/etc/supervisor/supervisord.conf

[unix_http_server]
file=/var/run/supervisor.sock
chmod=0700

[supervisord]
logfile=/var/log/supervisor/supervisord.log
pidfile=/var/run/supervisord.pid
childlogdir=/var/log/supervisor           
logfile_maxbytes=50MB                           
logfile_backups=5                               
loglevel=error                                  

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock

[include]
files = /etc/supervisor/conf.d/*.conf

/etc/supervisor/conf.d/tornado.conf

[program:tornado]
numprocs = 4
numprocs_start = 8000
process_name = %(program_name)s_%(process_num)02d
directory=/home/username/webapp
environment=PATH="/home/username/.virtualenvs/tornado_env/bin"
command = python server.py --port=%(process_num)s
user=www-data
redirect_stderr=true
stdout_logfile=/var/log/webapp/%(program_name)s_%(process_num)02d.log
loglevel=info
commented

Running a Debian-like distribution and installed supervisor via apt-get.

Please report this to the maintainers of the Debian package you installed with apt-get. The Debian packages are created and maintained by people that are not part of the Supervisor project. Those packages use their own init scripts. Those init scripts are not provided by the Supervisor project itself. The developers of the Debian package would be the only ones that could provide support and make fixes to their scripts.

This repository contains various user-contributed init scripts for Supervisor. Since you are not using one of these scripts, I'll close this issue.

commented

@mnaberez And what if the only difference between the Ubuntu version here and the Debian package version I have is the status command? The start, stop, and restart are the same.

commented

The example Ubuntu init script in this repository was submitted by a user and is just copied from Ubuntu itself. The best place to get support or fixes for the package you installed with apt-get is to contact the Debian or Ubuntu maintainers responsible for the package.

Per the README, the scripts in this repository are user-contributed and the Supervisor maintainers can't provide support. We are just collecting them in one place at the request of users.

commented

@mnaberez Okay. I've now determined the Debian package nor the initscript is the problem.

Using the same configs as previously stated.

$ which supervisord
/usr/bin/supervisord
$ sudo apt-get purge supervisor 

$ sudo pip install supervisor
$ which supervisord
/usr/local/bin/supervisord

$ sudo /usr/local/bin/supervisord -c /etc/supervisor/supervisor.conf
$ ps -aux | grep python
root       546  0.1  2.6  16104  9644 ?        Ss   01:11   0:00 /usr/bin/python /usr/local/bin/supervisord -c /etc/supervisor/supervisor.conf
www-data   547  1.2  3.9  17988 14176 ?        S    01:11   0:02 python server.py --port=8002
www-data   548  1.2  3.8  17988 14120 ?        S    01:11   0:02 python server.py --port=8003
www-data   549  1.2  3.9  17988 14136 ?        S    01:11   0:02 python server.py --port=8000
www-data   550  1.2  3.9  17988 14196 ?        S    01:11   0:02 python server.py --port=8001
www-data   559  0.0  3.0  17988 11156 ?        S    01:12   0:00 python server.py --port=8003
www-data   560  0.0  3.0  17988 11220 ?        S    01:12   0:00 python server.py --port=8000
www-data   561  0.0  3.0  17988 11172 ?        S    01:12   0:00 python server.py --port=8002
www-data   562  0.0  3.0  17988 11188 ?        S    01:12   0:00 python server.py --port=8001
commented

If I understand correctly, you expected supervisord to spawn 4 processes but you see more than that in the ps output.

Minimal supervisord.conf:

[supervisord]
loglevel=info

[inet_http_server]
port = 127.0.0.1:9001

[supervisorctl]
serverurl = http://127.0.0.1:9001

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[program:cat]
command = /bin/cat
numprocs = 4
numprocs_start = 8000
process_name = %(program_name)s_%(process_num)02d
program = /bin/cat

Start supervisord in the foreground with that config:

$ supervisord --version
3.2.0
$ supervisord -n -c /path/to/supervisord.conf

Four processes are spawned:

2015-12-23 23:30:35,273 INFO Increased RLIMIT_NOFILE limit to 1024
2015-12-23 23:30:35,285 INFO RPC interface 'supervisor' initialized
2015-12-23 23:30:35,285 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2015-12-23 23:30:35,285 INFO supervisord started with pid 65707
2015-12-23 23:30:36,291 INFO spawned: 'cat_8003' with pid 65710
2015-12-23 23:30:36,293 INFO spawned: 'cat_8002' with pid 65711
2015-12-23 23:30:36,295 INFO spawned: 'cat_8001' with pid 65712
2015-12-23 23:30:36,297 INFO spawned: 'cat_8000' with pid 65713
2015-12-23 23:30:37,302 INFO success: cat_8003 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-12-23 23:30:37,302 INFO success: cat_8002 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-12-23 23:30:37,302 INFO success: cat_8001 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-12-23 23:30:37,302 INFO success: cat_8000 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I watched the log while doing various combinations of supervisorctl stop, start, and status. I never saw supervisord spawn more than 4, and I never saw it orphan any processes.

I think if the above works for you, then the orphaned processes you see are probably being caused by one of the following:

  • supervisord received SIGKILL and thus exited immediately, orphaning its children.
  • Your program python server.py forks its own children (try pstree to check this). If supervisord sent SIGKILL to your program, and you did not set killasgroup=true, then your program exited immediately and orphaned its children.
commented

You were correct about my python server.py forking its own children (I didn't have pstree to see this, so had to search the Python Tornado Documentation for the example I took). So, yeah, apparently one server process is started per CPU, which doesn't make any sense on my single core Raspberry Pi, but whatever.

Adding stopasgroup=true does seem to fix the unmanaged child processes.

Thank you very much.

Another question, though. From your minimal conf, I see you use [inet_http_server]. I am trying to use [unix_http_server] since I don't need the Web UI.

I believe I have the correct configuration between [unix_http_server] and [supervisorctl] as I was able to successfully use the supervisorctl command via a socket before I installed the lastet supervisor from pip.

When I run supervisorctl now, all I get is http://localhost:9001 refused connection. If I keep [inet_http_server], it works and I can see the page from http://localhost:9001, but as I said, I don't need it. So, I removed [inet_http_server], added [unix_http_server], and set the appropriate values.

My question is if I replaced [inet_http_server] with [unix_http_server] and my [supervisorctl] section has serverurl = unix:///var/run/supervisord.sock, why does it try to connect over http and how do I properly setup the socket?

commented

When I run supervisorctl now, all I get is http://localhost:9001 refused connection. If I keep [inet_http_server], it works and I can see the page from http://localhost:9001, but as I said, I don't need it. So, I removed [inet_http_server], added [unix_http_server], and set the appropriate values.

If you ran supervisorctl with no arguments, then supervisorctl is searching for a config file. If it doesn't find a config file, it will still run, but will use built-in defaults. To make sure you are running supervisorctl with the correct config file, specify the path with -c:

$ supervisorctl -c /etc/supervisor/supervisord.conf

You said you changed from using a Debian-packaged version (apt-get install) of Supervisor to the PyPI version (pip install). I have seen Debian packaged versions where the Debian maintainers have modified supervisord and supervisorctl to additionally search for /etc/supervisor/supervisord.conf. No PyPI version of Supervisor has ever searched for this path as of Supervisor 3.2.0 (see the page linked above). If your config file is still in /etc/supervisor/supervisord.conf, that could explain the behavior.