Jupyterhub won't start after deployment

[Marcel Boersma]

I try to deploy the jupyterhub-deploy-teaching master version. Every step of deploy.yml executes without error except for start_jupyterhub.

At first it threw the following error:

TASK [start_jupyterhub : restart jupyterhub with supervisor] *********************************************************************************************************************************************************************************
fatal: [www.datascienceauditor.nl]: FAILED! => {"changed": false, "failed": true, "msg": "jupyterhub: ERROR (not running)\njupyterhub: ERROR (spawn error)\n"}

The error logs in /var/log/jupyterhub/jupyterhub.log mention that jupyterhub has trouble adding users that already exist:

[E 2017-09-13 10:51:20.557 JupyterHub app:1527]
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 1524, in launch_instance_async
        yield self.initialize(argv)
      File "/opt/conda/lib/python3.5/types.py", line 179, in throw
        return self.__wrapped.throw(tp, *rest)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 1311, in initialize
        yield self.init_groups()
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 935, in init_groups
        raise ValueError("Username %r is not in whitelist" % username)
    ValueError: Username 'instructor2' is not in whitelist

I removed the sqlite database from /srv/jupyterhub/jupyterhub.sqlite which resolves the problem of the ansible-playbook execution. Now everything executes fine without any errors. Nevertheless, when visiting my domain I get the nginx 502 Bad Gateway message so it seems that jupyterhub is not running.

If I ssh into the host machine and run jupyterhub it fires up jupyterhub and when I visit the domain I can see the login screen. But firing up jupyterhub this way uses some default configuration and not the one deployed by the ansible-playbook. Running jupyterhub with the path to the config file gives

sudo /opt/conda/bin/jupyterhub --config =/etc/jupyterhub/jupyterhub_config.py
[I 2017-09-13 11:03:14.112 JupyterHub app:745] Writing cookie_secret to /etc/jupyterhub/jupyterhub_cookie_secret
[W 2017-09-13 11:03:14.463 JupyterHub app:365]
    Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
    Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.

[W 2017-09-13 11:03:14.495 JupyterHub app:864] No admin users, admin interface will be unavailable.
[W 2017-09-13 11:03:14.495 JupyterHub app:865] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2017-09-13 11:03:14.496 JupyterHub app:892] Not using whitelist. Any authenticated user will be allowed.
[I 2017-09-13 11:03:14.511 JupyterHub app:1453] Hub API listening on http://127.0.0.1:8081/hub/
[W 2017-09-13 11:03:14.515 JupyterHub app:1174] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2017-09-13 11:03:14.515 JupyterHub app:1176] Starting proxy @ http://*:8000/
[E 2017-09-13 11:03:14.517 JupyterHub app:1184] Failed to find proxy ['configurable-http-proxy']
    The proxy can be installed with `npm install -g configurable-http-proxy`

So I'm a bit lost right now, is there anyway to resolve this problem?

[Andrea Zonca]

it looks like a PATH problem. try to change user to root with sudo su, then check that /opt/conda/bin is in your path, then try which configurable-http-proxy to check if that is available

[Andrea Zonca]

also, do you want to use nbgrader? if so, instructor1 and the other users needs to be created on the server before running ansible

[Marcel Boersma]

@zonca thanks for the quick response. I followed your suggestions which resulted in the following output

ubuntu@www:~$ sudo su
root@www:/home/ubuntu# echo $PATH
/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
root@www:/home/ubuntu# which configurable-http-proxy
/opt/conda/bin/configurable-http-proxy
root@www:/home/ubuntu#

The last command is a manual start of the jupyterhub server not the one initiated by the ansible script.
I added the users instructor1 instructor2 grader1 grader2_ before running the sensible script. I realized that the whitelisted users are instructor grader student1 student2 and I redeployed using the ansible-playbook deploy.yml command. However, it still shows the Bad gateway 502 warning.
Checking the /var/log/jupyterhub/jupyterhub.log showed:

[I 2017-09-13 17:58:43.799 JupyterHub app:724] Loading cookie_secret from /srv/jupyterhub/cookie_secret
[E 2017-09-13 17:58:44.225 JupyterHub app:1527]
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 1524, in launch_instance_async
        yield self.initialize(argv)
      File "/opt/conda/lib/python3.5/types.py", line 179, in throw
        return self.__wrapped.throw(tp, *rest)
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 1311, in initialize
        yield self.init_groups()
      File "/opt/conda/lib/python3.5/site-packages/jupyterhub/app.py", line 935, in init_groups
        raise ValueError("Username %r is not in whitelist" % username)
    ValueError: Username 'instructor1' is not in whitelist

Apparently it is due to the user adding issue. I now completed the whitelist and added all the users.
The issue is that in /host_vars/hostname.example on line 16-25 usernames are specified that are different from the ones on line 183-195 used for nbgrader. Thanks for the feedback 👍 I might submit a small merge request resolving the mentioned problem.

[Andrea Zonca]

if now it works please close the issue, a Pull Request is very welcome