Stuck Starting Up
hakasapl opened this issue · comments
Bug description
When I try to spawn a JupyterLab instance using the Slurm spawner, the job launches according to slurm, however, the jupyterhub window just stays in the "Your server is starting up" page.
Expected behaviour
Allow me to enter the JupyterLab session that is launched
Actual behaviour
Stuck in launch screen.
How to reproduce
- Open JupyterLab.
- Launch session
Your personal set up
- OS: ubuntu 20.04
- Version(s): JupyterHub=1.3 JupyterLab=3.0.4 Batchspawner=1.0.1
There is a firewall between the Hub and the Lab, but I've temporarily disabled it for testing purposes. It seems like JupyterLab can communicate with JupyterHub, because when I stop JupyterHub, JupyterLab starts spitting out connection refused messages - leading me to believe that it was communicating when it was up.
Maybe there is some batchspawner configuration I'm missing with the 1.0 update?
alembic @ file:///home/conda/feedstock_root/build_artifacts/alembic_1599849174714/work
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1610315732543/work/dist
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1605217007488/work
async-generator==1.10
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1605083924122/work
Babel @ file:///home/conda/feedstock_root/build_artifacts/babel_1605182336601/work
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backports.functools-lru-cache==1.6.1
batchspawner==1.0.1
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1600454382015/work
blinker==1.4
brotlipy==0.7.0
certifi==2020.12.5
certipy==0.1.3
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1606601120025/work
chardet @ file:///home/conda/feedstock_root/build_artifacts/chardet_1610093492116/work
conda==4.9.2
conda-package-handling @ file:///home/conda/feedstock_root/build_artifacts/conda-package-handling_1602876790736/work
cryptography @ file:///home/conda/feedstock_root/build_artifacts/cryptography_1610338668572/work
decorator==4.4.2
defusedxml==0.6.0
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1602701735325/work/dist/entrypoints-0.3-py2.py3-none-any.whl
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1593328102638/work
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1610355162947/work
ipykernel @ file:///home/conda/feedstock_root/build_artifacts/ipykernel_1607737221803/work/dist/ipykernel-5.4.2-py3-none-any.whl
ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1609621631159/work
ipython-genutils==0.2.0
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1605054524051/work
jhub-remote-user-authenticator @ git+https://github.com/UMass-RC/jhub_remote_user_authenticator@41ef0fd940db8f4175d2b5d485ecea33c4bf36da
Jinja2==2.11.2
json5 @ file:///home/conda/feedstock_root/build_artifacts/json5_1600692310011/work
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1602551949684/work
jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1610375432619/work
jupyter-core @ file:///home/conda/feedstock_root/build_artifacts/jupyter_core_1605735002943/work
jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1610137539342/work
jupyter-telemetry @ file:///home/conda/feedstock_root/build_artifacts/jupyter_telemetry_1605173804246/work
jupyterhub @ file:///home/conda/feedstock_root/build_artifacts/jupyterhub-feedstock_1607688168448/work
jupyterlab @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_1610500889619/work
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work
jupyterlab-server @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_server_1610492632103/work
Mako @ file:///home/conda/feedstock_root/build_artifacts/mako_1595925083607/work
mamba @ file:///home/conda/feedstock_root/build_artifacts/mamba_1608134852777/work
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1610127590546/work
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1610112882083/work
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1610352513187/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1602859080374/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert_1605401842418/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1602732862338/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1605195931949/work
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1608772651579/work
oauthlib==3.1.0
optionsspawner @ git+https://github.com/UMass-RC/jupyterhub-options-spawner.git@8f21eb63282ce34e6a331b5f1c09f6d1cc9178e1
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1607785313469/work
pamela==1.0.0
pandocfilters @ file:///tmp/build/80754af9/pandocfilters_1605120906940/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1595548966091/work
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602535621604/work
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1605543085815/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1610127285153/work
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pycosat @ file:///home/conda/feedstock_root/build_artifacts/pycosat_1610094799048/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1593275161868/work
pycurl==7.43.0.6
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1607352093725/work
PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1609976754451/work
pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1608055815057/work
pyparsing==2.4.7
pyrsistent @ file:///home/conda/feedstock_root/build_artifacts/pyrsistent_1610146801554/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1610291451001/work
python-dateutil==2.8.1
python-editor==1.0.4
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1602545356084/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1608904108784/work
pyzmq==20.0.0
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1608156231189/work
ruamel-yaml-conda @ file:///home/conda/feedstock_root/build_artifacts/ruamel_yaml_1610408217628/work
ruamel.yaml @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml_1610291374928/work
ruamel.yaml.clib @ file:///home/conda/feedstock_root/build_artifacts/ruamel.yaml.clib_1610146844134/work
Send2Trash==1.5.0
six @ file:///home/conda/feedstock_root/build_artifacts/six_1590081179328/work
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1610318319523/work
SQLAlchemy @ file:///home/conda/feedstock_root/build_artifacts/sqlalchemy_1610127208390/work
terminado @ file:///home/conda/feedstock_root/build_artifacts/terminado_1609794176566/work
testpath==0.4.4
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1610094708661/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1610419111160/work
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1605225765842/work
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work
webencodings==0.5.1
wrapspawner @ git+https://github.com/jupyterhub/wrapspawner@89324d46b2d49a7e640155376e7e25ff137c054d
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1603668650351/work
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
alembic 1.4.3 pyh9f0ad1d_0 conda-forge
anyio 2.0.2 py39hf3d152e_4 conda-forge
argon2-cffi 20.1.0 py39hbd71b63_2 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 20.3.0 pyhd3deb0d_0 conda-forge
babel 2.9.0 pyhd3deb0d_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.1 py_0 conda-forge
batchspawner 1.0.1 pypi_0 pypi
bleach 3.2.1 pyh9f0ad1d_0 conda-forge
blinker 1.4 py_1 conda-forge
brotlipy 0.7.0 py39h38d8fee_1001 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.17.1 h36c2ea0_0 conda-forge
ca-certificates 2020.12.8 h06a4308_0
certifi 2020.12.5 py39hf3d152e_1 conda-forge
certipy 0.1.3 py_0 conda-forge
cffi 1.14.4 py39he32792d_1 conda-forge
chardet 4.0.0 py39hf3d152e_1 conda-forge
conda 4.9.2 py39hf3d152e_0 conda-forge
conda-package-handling 1.7.2 py39h38d8fee_0 conda-forge
configurable-http-proxy 4.2.1 node12_hbce0df8_0 conda-forge
cryptography 3.3.1 py39h3da14fd_1 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
entrypoints 0.3 py39hde42818_1002 conda-forge
icu 68.1 h58526e2_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 3.4.0 py39hf3d152e_0 conda-forge
importlib_metadata 3.4.0 hd8ed1ab_0 conda-forge
ipykernel 5.4.2 py39hef51801_0 conda-forge
ipython 7.19.0 py39hef51801_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
jedi 0.17.2 py39hf3d152e_1 conda-forge
jhub-remote-user-authenticator 0.1.0 pypi_0 pypi
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
json5 0.9.5 pyh9f0ad1d_0 conda-forge
jsonschema 3.2.0 py_2 conda-forge
jupyter_client 6.1.11 pyhd8ed1ab_1 conda-forge
jupyter_core 4.7.0 py39hf3d152e_0 conda-forge
jupyter_server 1.2.1 py39hf3d152e_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub 1.3.0 py39hf3d152e_0 conda-forge
jupyterhub-base 1.3.0 py39hf3d152e_0 conda-forge
jupyterlab 3.0.4 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_server 2.1.2 pyhd8ed1ab_0 conda-forge
krb5 1.17.2 h926e7f8_0 conda-forge
ld_impl_linux-64 2.35.1 hea4e1c9_1 conda-forge
libarchive 3.5.1 h3f442fb_1 conda-forge
libcurl 7.71.1 hcdd3856_8 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.3 h58526e2_2 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libnghttp2 1.41.0 h8cfc5f6_2 conda-forge
libsodium 1.0.18 h516909a_1 conda-forge
libsolv 0.7.16 h8b12597_0 conda-forge
libssh2 1.9.0 hab1572f_5 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
libuv 1.40.0 hd18ef5c_0 conda-forge
libxml2 2.9.10 h72842e0_3 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
mako 1.1.3 pyh9f0ad1d_0 conda-forge
mamba 0.7.6 py39heb6f48a_0 conda-forge
markupsafe 1.1.1 py39h3811e60_3 conda-forge
mistune 0.8.4 py39h3811e60_1003 conda-forge
nbclassic 0.2.6 pyhd8ed1ab_0 conda-forge
nbclient 0.5.1 py_0 conda-forge
nbconvert 6.0.7 py39hf3d152e_3 conda-forge
nbformat 5.0.8 py_0 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
nest-asyncio 1.4.3 pyhd8ed1ab_0 conda-forge
nodejs 12.19.0 hfa01f41_2 conda-forge
notebook 6.1.6 py39hf3d152e_0 conda-forge
oauthlib 3.1.0 py_0
openssl 1.1.1i h7f98852_0 conda-forge
optionsspawner 0.1.0 pypi_0 pypi
packaging 20.8 pyhd3deb0d_0 conda-forge
pamela 1.0.0 py_0 conda-forge
pandoc 2.11.3.2 h7f98852_0 conda-forge
pandocfilters 1.4.3 py39h06a4308_1
parso 0.7.1 pyh9f0ad1d_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py39hde42818_1002 conda-forge
pip 20.3.3 pyhd8ed1ab_0 conda-forge
prometheus_client 0.9.0 pyhd3deb0d_0 conda-forge
prompt-toolkit 3.0.10 pyha770c72_0 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pycosat 0.6.3 py39h3811e60_1006 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pycurl 7.43.0.6 py39h72e3413_1 conda-forge
pygments 2.7.3 pyhd8ed1ab_0 conda-forge
pyjwt 2.0.0 pyhd8ed1ab_0 conda-forge
pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyrsistent 0.17.3 py39h3811e60_2 conda-forge
pysocks 1.7.1 py39hf3d152e_3 conda-forge
python 3.9.1 hffdb5ce_3_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-editor 1.0.4 py_0 conda-forge
python-json-logger 2.0.1 pyh9f0ad1d_0 conda-forge
python_abi 3.9 1_cp39 conda-forge
pytz 2020.5 pyhd8ed1ab_0 conda-forge
pyzmq 20.0.0 py39hea8fd45_1 conda-forge
readline 8.0 he28a2e2_2 conda-forge
reproc 14.2.1 h36c2ea0_0 conda-forge
reproc-cpp 14.2.1 h58526e2_0 conda-forge
requests 2.25.1 pyhd3deb0d_0 conda-forge
ruamel.yaml 0.16.12 py39h3811e60_2 conda-forge
ruamel.yaml.clib 0.2.2 py39h3811e60_2 conda-forge
ruamel_yaml 0.15.80 py39h3811e60_1003 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.6.0 py39hf3d152e_3 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
sniffio 1.2.0 py39hf3d152e_1 conda-forge
sqlalchemy 1.3.22 py39h3811e60_1 conda-forge
sqlite 3.34.0 h74cdb3f_0 conda-forge
terminado 0.9.2 py39hf3d152e_0 conda-forge
testpath 0.4.4 py_0 conda-forge
tk 8.6.10 h21135ba_1 conda-forge
tornado 6.1 py39h3811e60_1 conda-forge
tqdm 4.56.0 pyhd8ed1ab_0 conda-forge
traitlets 5.0.5 py_0 conda-forge
tzdata 2020f he74cb21_0 conda-forge
urllib3 1.26.2 pyhd8ed1ab_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.36.2 pyhd3deb0d_0 conda-forge
wrapspawner 1.0.0 pypi_0 pypi
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
zeromq 4.3.3 h58526e2_3 conda-forge
zipp 3.4.0 py_0 conda-forge
zlib 1.2.11 h516909a_1010 conda-forge
zstd 1.4.8 ha95c52a_1 conda-forge
Configuration
Slurm Spawner, slurm v 20.11.2# jupyterhub_config.py
import os
import importlib.machinery
import batchspawner
c.JupyterHub.log_level = 'DEBUG'
jhub_dir = "/srv/www/jhub"
slurm_config = importlib.machinery.SourceFileLoader('slurm_config',jhub_dir + '/slurm_config.py').load_module()
form_config = importlib.machinery.SourceFileLoader('form_config',jhub_dir + '/form_config.py').load_module()
c.JupyterHub.cleanup_servers = False
# Shibboleth Authenticator
c.JupyterHub.authenticator_class = 'jhub_remote_user_authenticator.remote_user_auth.RemoteUserLocalAuthenticator'
c.JupyterHub.spawner_class = 'optionsspawner.OptionsFormSpawner'
# This is what allows JHUB to not be at the root / folder
c.JupyterHub.bind_url = 'http://:8000/panel/jhub'
c.JupyterHub.hub_bind_url = 'http://:8081'
c.ConfigurableHTTPProxy.api_url = 'http://0.0.0.0:8001'
# Set notebook directory to the user's home directory
c.Spawner.notebook_dir = '~' # Default to home directory
#c.Spawner.default_url = '/lab' # Default to jupyterlab
# Set admin users TODO use sudo instead
c.Authenticator.admin_users = {'hsaplakoglu_umass_edu','jgriffin_umass_edu','aryamanagraw_umass_edu'}
c.JupyterHub.admin_access = True
c.Spawner.cmd=["jupyterhub-singleuser"]
c.Spawner.ip = '0.0.0.0'
c.Spawner.http_timeout = 300
c.Spawner.start_timeout = 3600
c.OptionsFormSpawner.child_class = 'batchspawner.SlurmSpawner'
c.SlurmSpawner.batch_script = slurm_config.batch_script
c.OptionsFormSpawner.form_fields = form_config.form_fields# slurm_config.py
batch_script="""#!/bin/bash
#SBATCH --partition={partition}
#SBATCH --time={runtime}
#SBATCH --ntasks-per-node={ntasks}
#SBATCH --cpus-per-task={ncpus}
#SBATCH --gpus={numgpus}
#SBATCH --output={homedir}/.jupyterhub-slurmspawner.log
#SBATCH --open-mode=append
#SBATCH --job-name=JupyterLab
#SBATCH --export={keepvars}
source /etc/profile
export MODULEPATH=$MODULEPATH:/modules/system
module load R
module load miniconda
conda activate jupyter-3
{cmd}
"""# form_config.py
from optionsspawner.forms import (
FormField,
TextInputField,
NumericalInputField,
CheckboxInputField,
SelectField
)
partition_select = SelectField('req_partition',
label='Select a partition',
attr_required=True,
choices=[
('cpu', "CPU"),
('gpu', "GPU"),
('cee_water_cjgleason', "CEE-Water (cjgleason)"),
('cee_water_casey', "CEE-Water (casey)"),
('cee_water_kandread', "CEE-Water (kandread)")
],
default='cpu'
)
runtime_input = TextInputField('req_runtime',
label='Runtime (HH:MM:SS) - Your instance is killed at the end of runtime',
attr_required=True,
attr_value='02:00:00'
#attr_pattern="[01]{1}[0-2]{1}:[0-5]{1}[0-9]{1}:[0-5]{1}[0-9]{1}"
)
ntasks_input = NumericalInputField('req_ntasks',
label="# Tasks per Node - For MPI Only, Leave at 1 if unsure",
attr_required=True,
attr_value=1,
attr_min=1,
attr_max=80
)
ncpus_input = NumericalInputField('req_ncpus',
label='# Cores/Threads per Task',
attr_required=True,
attr_value=1,
attr_min=1,
attr_max=80
)
ngpus_input = NumericalInputField('req_numgpus',
label='# GPUs - GPU Partitions Only',
attr_required=True,
attr_value=0
)
form_fields = [
partition_select,
runtime_input,
ntasks_input,
ncpus_input,
ngpus_input
]Logs
# JupyterHub Log
[D 2021-01-13 15:58:42.111 JupyterHub pages:252] Triggering spawn with supplied form options for hsaplakoglu_umass_edu
[D 2021-01-13 15:58:42.112 JupyterHub base:875] Initiating spawn for hsaplakoglu_umass_edu
[D 2021-01-13 15:58:42.112 JupyterHub base:879] 0/100 concurrent spawns
[D 2021-01-13 15:58:42.113 JupyterHub base:884] 0 active servers
[D 2021-01-13 15:58:42.154 JupyterHub user:602] Calling Spawner.start for hsaplakoglu_umass_edu
[I 2021-01-13 15:58:42.164 JupyterHub batchspawner:248] Spawner submitting job using sudo -E -u hsaplakoglu_umass_edu sbatch --parsable
[I 2021-01-13 15:58:42.164 JupyterHub batchspawner:249] Spawner submitted script:
#!/bin/bash
#SBATCH --partition=cpu
#SBATCH --time=02:00:00
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --gpus=0
#SBATCH --output=/home/hsaplakoglu_umass_edu/.jupyterhub-slurmspawner.log
#SBATCH --open-mode=append
#SBATCH --job-name=JupyterLab
#SBATCH --export=PATH,CONDA_DEFAULT_ENV,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,USER,HOME,SHELL
source /etc/profile
export MODULEPATH=$MODULEPATH:/modules/system
module load R
module load miniconda
conda activate jupyter-3
batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0 --notebook-dir=~
[I 2021-01-13 15:58:42.220 JupyterHub batchspawner:252] Job submitted. cmd: sudo -E -u hsaplakoglu_umass_edu sbatch --parsable output: 1332952
[D 2021-01-13 15:58:42.221 JupyterHub batchspawner:274] Spawner querying job: sudo -E -u hsaplakoglu_umass_edu squeue -h -j 1332952 -o '%T %B'
[D 2021-01-13 15:58:42.268 JupyterHub batchspawner:374] Job 1332952 still pending
[D 2021-01-13 15:58:42.772 JupyterHub batchspawner:274] Spawner querying job: sudo -E -u hsaplakoglu_umass_edu squeue -h -j 1332952 -o '%T %B'
[I 2021-01-13 15:58:43.118 JupyterHub log:181] 302 POST /panel/jhub/hub/spawn/hsaplakoglu_umass_edu -> /panel/jhub/hub/spawn-pending/hsaplakoglu_umass_edu (hsaplakoglu_umass_edu@::ffff:127.0.0.1) 1011.38ms
[I 2021-01-13 15:58:43.181 JupyterHub pages:402] hsaplakoglu_umass_edu is pending spawn
[I 2021-01-13 15:58:43.188 JupyterHub log:181] 200 GET /panel/jhub/hub/spawn-pending/hsaplakoglu_umass_edu (hsaplakoglu_umass_edu@::ffff:127.0.0.1) 21.05ms
[D 2021-01-13 15:58:45.659 JupyterHub base:283] Recording first activity for <APIToken('92e3...', user='hsaplakoglu_umass_edu')>
[I 2021-01-13 15:58:45.669 JupyterHub log:181] 200 POST /panel/jhub/hub/api/batchspawner (hsaplakoglu_umass_edu@10.100.10.5) 17.80ms
[I 2021-01-13 15:58:49.540 JupyterHub log:181] 200 GET /panel/jhub/hub/api (@10.100.10.5) 2.03ms
[D 2021-01-13 15:58:49.567 JupyterHub users:765] Activity for user hsaplakoglu_umass_edu: 2021-01-13T15:58:47.555958Z
[D 2021-01-13 15:58:49.568 JupyterHub users:783] Activity on server hsaplakoglu_umass_edu/: 2021-01-13T15:58:47.555958Z
[I 2021-01-13 15:58:49.575 JupyterHub log:181] 200 POST /panel/jhub/hub/api/users/hsaplakoglu_umass_edu/activity (hsaplakoglu_umass_edu@10.100.10.5) 19.23ms
[W 2021-01-13 15:58:52.114 JupyterHub base:1003] User hsaplakoglu_umass_edu is slow to start (timeout=10)
[D 2021-01-13 15:59:34.940 JupyterHub users:770] Not updating activity for <User(hsaplakoglu_umass_edu 1/1 running)>: 2021-01-13T15:17:32.248437Z < 2021-01-13T15:58:47.555958Z
[D 2021-01-13 15:59:34.940 JupyterHub users:791] Not updating server activity on hsaplakoglu_umass_edu/: 2021-01-13T15:17:32.248437Z < 2021-01-13T15:58:47.555958Z
[I 2021-01-13 15:59:34.941 JupyterHub log:181] 200 POST /panel/jhub/hub/api/users/hsaplakoglu_umass_edu/activity (hsaplakoglu_umass_edu@10.100.10.3) 17.89ms
[I 2021-01-13 16:02:00.192 JupyterHub log:181] 302 GET /panel/jhub -> /panel/jhub/hub/ (@::ffff:127.0.0.1) 3.00ms
[W 2021-01-13 16:02:00.425 JupyterHub base:390] Invalid or expired cookie token
# JupyterLab Log
slurmstepd: error: couldn't chdir to `/srv/www/jhub': No such file or directory: going to /tmp instead
[W 2021-01-13 05:27:25.579 SingleUserLabApp configurable:190] Config option `default_url` not recognized by `SingleUserLabApp`.
[W 2021-01-13 05:27:25.582 SingleUserLabApp configurable:190] Config option `default_url` not recognized by `SingleUserLabApp`.
[W 2021-01-13 05:27:25.584 SingleUserLabApp configurable:190] Config option `default_url` not recognized by `SingleUserLabApp`.
[W 2021-01-13 05:27:25.584 SingleUserLabApp configurable:190] Config option `default_url` not recognized by `SingleUserLabApp`.
[I 2021-01-13 05:27:26.130 LabApp] JupyterLab extension loaded from /modules/apps/miniconda/4.8.3/envs/jupyter/lib/python3.8/site-packages/jupyterlab
[I 2021-01-13 05:27:26.130 LabApp] JupyterLab application directory is /modules/apps/miniconda/4.8.3/envs/jupyter/share/jupyter/lab
[I 2021-01-13 05:27:26.144 SingleUserLabApp.mamba_gator handlers:513] Server extension enabled
[I 2021-01-13 05:27:26.144 SingleUserLabApp mixins:557] Starting jupyterhub-singleuser server version 1.3.0
[I 2021-01-13 05:27:26.152 SingleUserLabApp notebookapp:2257] Serving notebooks from local directory: /home/hsaplakoglu_umass_edu
[I 2021-01-13 05:27:26.152 SingleUserLabApp notebookapp:2257] Jupyter Notebook 6.1.6 is running at:
[I 2021-01-13 05:27:26.152 SingleUserLabApp notebookapp:2257] http://node3:36327/
[I 2021-01-13 05:27:26.152 SingleUserLabApp notebookapp:2258] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2021-01-13 05:27:26.165 SingleUserLabApp mixins:537] Updating Hub with activity every 300 seconds
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋
Welcome to the Jupyter community! 🎉
This issue seems to be because of the spawner I'm using. If I use a wrapspawner (in my case optionsformspawner, although I also tried profile spawner in my troubleshooting), it doesn't work. If I batch directly using batchspawner, it works just fine.
Is there a configuration change that needs to be made for wrapspawner-based extensions?
I've had a similar problem to this and I've hacked a bit on the piece of batchspawner that allows me to accomplish this. Wasn't straightforward for a bit. I'm not going to say it's a good fix but you can look here: https://github.com/NCAR/batchspawner/blob/a31e9e2cd6803c3b940745cc7495c01a3c23badd/batchspawner/api.py#L22.
Effectively, I found that when nested below wrapspawner in our case, batchspawner on the client side was indeed posting the port assigned, but was being applied to the wrong parent spawner in this case, wrapspawner instead of batchspawner.
Thanks for sharing this @jbaksta. It also worked for me. I also had to set the following environment variable to make it work JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}
else I get 404 errors in my apache log.
@jbaksta must we create a pull request for this problem?
Thanks for sharing this @jbaksta. It also worked for me. I also had to set the following environment variable to make it work
JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}else I get 404 errors in my apache log.
How to set this up, please?
We believe this has been resolved by jupyterhub/wrapspawner#51 which is included in the most recent Wrapspawner release.