v4.1.5: oauth redirect loop
MetRonnie opened this issue · comments
Bug description
Hello, here at Cylc we're having an issue with Jupyterhub 4.1.5. When launching the hub and trying to access our web app, I get 403 errors for the JS and CSS files linked in the index.html
of the app. This seems related to a series of bugfix releases from 4.1.0-4.1.5 regarding XSRF cookies - @minrk do you have any ideas as to what's going on from a quick look at this snippet?
Sanitised log snippet:
[I CylcHubApp log:192] 302 GET /user/jbloggs/cylc/assets/index-BRxHbm_z.css -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.54ms
[I CylcHubApp log:192] 302 GET /user/jbloggs/cylc/assets/index-Cs-VwyI1.js -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 2.02ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] -> /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 122.37ms
[W CylcHubApp auth:1494] oauth state argument 'e1f2g3h4' != cookie jupyterhub-user-jbloggs-oauth-state='e567fgh'
[W CylcHubApp web:1873] 403 GET /user/jbloggs/oauth_callback?code=E789HIJ&state=e1f2g3h4 (::ffff:XX.XXX.XXX.XX): oauth state does not match. Try logging in again.
[W CylcHubApp log:192] 403 GET /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (@::ffff:XX.XXX.XXX.XX) 44.85ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] -> /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 127.54ms
[W CylcHubApp auth:1494] oauth state argument 'e567fgh' != cookie jupyterhub-user-jbloggs-oauth-state=None
[W CylcHubApp web:1873] 403 GET /user/jbloggs/oauth_callback?code=F4H5IK6&state=e567fgh (::ffff:XX.XXX.XXX.XX): oauth state does not match. Try logging in again.
[W CylcHubApp log:192] 403 GET /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (@::ffff:XX.XXX.XXX.XX) 2.50ms
I also get ERR_TOO_MANY_REDIRECTS
in the browser.
How to reproduce
- Install cylc-uiserver e.g.
pip install cylc-uiserver[hub]==1.4.4 jupyterhub==4.1.5
andconfigurable-http-proxy
via conda or npm - Run:
cylc hub
, or equivalently:CYLC_HUB_VERSION=1.4.4 jupyterhub --config ~/.conda/envs/envname/lib/python3.9/site-packages/cylc/uiserver/jupyterhub_config.py
- Open the hub in the browser and log in
Expected behaviour
In Jupyterhub 4.1.4, the app loads after logging in.
Actual behaviour
403 error as described above.
Your personal set up
- OS: RHEL 7.9
- Version(s): jupyterhub 4.1.5, python 3.9 & python 3.11 tested
Full environment
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alembic 1.13.1 pypi_0 pypi
aniso8601 7.0.0 pypi_0 pypi
annotated-types 0.6.0 pypi_0 pypi
ansimarkup 2.1.0 pypi_0 pypi
anyio 4.3.0 pypi_0 pypi
argon2-cffi 23.1.0 pypi_0 pypi
argon2-cffi-bindings 21.2.0 pypi_0 pypi
arrow 1.3.0 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
async-generator 1.10 pypi_0 pypi
async-lru 2.0.4 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
babel 2.14.0 pypi_0 pypi
beautifulsoup4 4.12.3 pypi_0 pypi
bleach 6.1.0 pypi_0 pypi
bzip2 1.0.8 hd590300_5 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
certifi 2024.2.2 pypi_0 pypi
certipy 0.1.3 pypi_0 pypi
cffi 1.16.0 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
colorama 0.4.6 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
cryptography 42.0.5 pypi_0 pypi
cylc-flow 8.2.5 pypi_0 pypi
cylc-uiserver 1.4.4 pypi_0 pypi
debugpy 1.8.1 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
exceptiongroup 1.2.1 pypi_0 pypi
executing 2.0.1 pypi_0 pypi
fastjsonschema 2.19.1 pypi_0 pypi
fqdn 1.5.1 pypi_0 pypi
graphene 2.1.9 pypi_0 pypi
graphene-tornado 2.6.1 pypi_0 pypi
graphql-core 2.3.2 pypi_0 pypi
graphql-relay 2.0.1 pypi_0 pypi
graphql-ws 0.4.4 pypi_0 pypi
greenlet 3.0.3 pypi_0 pypi
h11 0.14.0 pypi_0 pypi
httpcore 1.0.5 pypi_0 pypi
httpx 0.27.0 pypi_0 pypi
idna 3.7 pypi_0 pypi
importlib-metadata 7.1.0 pypi_0 pypi
ipykernel 6.29.4 pypi_0 pypi
ipython 8.18.1 pypi_0 pypi
isoduration 20.11.0 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.0.3 pypi_0 pypi
json5 0.9.25 pypi_0 pypi
jsonpointer 2.4 pypi_0 pypi
jsonschema 4.21.1 pypi_0 pypi
jsonschema-specifications 2023.12.1 pypi_0 pypi
jupyter-client 8.6.1 pypi_0 pypi
jupyter-core 5.7.2 pypi_0 pypi
jupyter-events 0.10.0 pypi_0 pypi
jupyter-lsp 2.2.5 pypi_0 pypi
jupyter-server 2.14.0 pypi_0 pypi
jupyter-server-terminals 0.5.3 pypi_0 pypi
jupyter-telemetry 0.1.0 pypi_0 pypi
jupyterhub 4.1.5 pypi_0 pypi
jupyterlab 4.1.6 pypi_0 pypi
jupyterlab-pygments 0.3.0 pypi_0 pypi
jupyterlab-server 2.27.1 pypi_0 pypi
ld_impl_linux-64 2.40 h55db66e_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 hc881cc4_6 conda-forge
libgomp 13.2.0 hc881cc4_6 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsqlite 3.45.3 h2797004_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
mako 1.3.3 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib-inline 0.1.7 pypi_0 pypi
metomi-isodatetime 1!3.1.0 pypi_0 pypi
mistune 3.0.2 pypi_0 pypi
nbclient 0.10.0 pypi_0 pypi
nbconvert 7.16.3 pypi_0 pypi
nbformat 5.10.4 pypi_0 pypi
ncurses 6.4.20240210 h59595ed_0 conda-forge
nest-asyncio 1.6.0 pypi_0 pypi
notebook-shim 0.2.4 pypi_0 pypi
oauthlib 3.2.2 pypi_0 pypi
openssl 3.2.1 hd590300_1 conda-forge
overrides 7.7.0 pypi_0 pypi
packaging 24.0 pypi_0 pypi
pamela 1.1.0 pypi_0 pypi
pandocfilters 1.5.1 pypi_0 pypi
parso 0.8.4 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
platformdirs 4.2.1 pypi_0 pypi
prometheus-client 0.20.0 pypi_0 pypi
promise 2.3 pypi_0 pypi
prompt-toolkit 3.0.43 pypi_0 pypi
protobuf 4.21.12 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pycparser 2.22 pypi_0 pypi
pydantic 2.7.1 pypi_0 pypi
pydantic-core 2.18.2 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pyopenssl 24.1.0 pypi_0 pypi
python 3.9.19 h0755675_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python-json-logger 2.0.7 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 26.0.2 pypi_0 pypi
readline 8.2 h8228510_1 conda-forge
referencing 0.35.0 pypi_0 pypi
requests 2.31.0 pypi_0 pypi
rfc3339-validator 0.1.4 pypi_0 pypi
rfc3986-validator 0.1.1 pypi_0 pypi
rpds-py 0.18.0 pypi_0 pypi
ruamel-yaml 0.18.6 pypi_0 pypi
ruamel-yaml-clib 0.2.8 pypi_0 pypi
rx 1.6.3 pypi_0 pypi
send2trash 1.8.3 pypi_0 pypi
setuptools 69.5.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
sniffio 1.3.1 pypi_0 pypi
soupsieve 2.5 pypi_0 pypi
sqlalchemy 2.0.29 pypi_0 pypi
stack-data 0.6.3 pypi_0 pypi
terminado 0.18.1 pypi_0 pypi
tinycss2 1.3.0 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
tomli 2.0.1 pypi_0 pypi
tornado 6.4 pypi_0 pypi
traitlets 5.14.3 pypi_0 pypi
types-python-dateutil 2.9.0.20240316 pypi_0 pypi
typing-extensions 4.11.0 pypi_0 pypi
tzdata 2024a h0c530f3_0 conda-forge
uri-template 1.3.0 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
urwid 2.6.11 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
webcolors 1.13 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
websocket-client 1.8.0 pypi_0 pypi
werkzeug 0.12.2 pypi_0 pypi
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zipp 3.18.1 pypi_0 pypi
Configuration
This:
https://github.com/cylc/cylc-uiserver/blob/1.4.4/cylc/uiserver/jupyterhub_config.py
which is used to load this:
import os
from pathlib import Path
import re
import sys
# -- Jupyter Hub Config
os.environ["no_proxy"]="127.0.0.1"
# Specify the location of all the JupyterHub runtime files
RUNTIME_PATH = Path('~/.cylc/uiserver').expanduser()
c.JupyterHub.cookie_secret_file = f'{RUNTIME_PATH / "cookie_secret"}'
c.JupyterHub.db_url = f'{RUNTIME_PATH / "jupyterhub.sqlite"}'
c.ConfigurableHTTPProxy.pid_file = f'{RUNTIME_PATH / "jupyterhub-proxy.pid"}'
# Create a self signed certificate if certificate directory not found
CERT_PATH = Path(RUNTIME_PATH, "cert")
if not CERT_PATH.exists():
CERT_PATH.mkdir(parents=True)
from subprocess import Popen
proc = Popen([
# path to the openssl executable in this environment
re.sub(r'python(\d[\.\d]*)?$', 'openssl', sys.executable),
'req',
'--x509',
'--nodes',
'--days=3650',
'--newkey=rsa:2048',
'--subj=/O=Cylc',
f'--keyout={CERT_PATH / "self_signed.key"}',
f'--out={CERT_PATH / "self_signed.crt"}'
])
if proc.wait():
raise Exception('Could not create certificate')
# Use self signed certificate by default
c.JupyterHub.ssl_cert = f'{CERT_PATH / "self_signed.crt"}'
c.JupyterHub.ssl_key = f'{CERT_PATH / "self_signed.key"}'
# -- Cylc UIS config
c.UIServer.scan_interval = 60 # PT60S
Logs
[I JupyterHub app:2885] Running JupyterHub version 4.1.5
[I JupyterHub app:2915] Using Authenticator: jupyterhub.auth.PAMAuthenticator-4.1.5
[I JupyterHub app:2915] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-4.1.5
[I JupyterHub app:2915] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-4.1.5
[I JupyterHub app:1683] Loading cookie_secret from ~/jbloggs/.cylc/uiserver/cookie_secret
[I JupyterHub proxy:557] Generating new CONFIGPROXY_AUTH_TOKEN
[I JupyterHub app:2005] Not using allowed_users. Any authenticated user will be allowed.
[I JupyterHub app:2954] Initialized 0 spawners in 0.007 seconds
[I JupyterHub metrics:279] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I JupyterHub metrics:279] Found 1 active users in the last ActiveUserPeriods.seven_days
[I JupyterHub metrics:279] Found 1 active users in the last ActiveUserPeriods.thirty_days
[I JupyterHub proxy:751] Starting proxy @ https://:8000
[ConfigProxy] info: Proxying https://*:8000 to (no default)
[ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[ConfigProxy] info: 200 GET /api/routes
[I JupyterHub app:3204] Hub API listening on http://127.0.0.1:8081/hub/
[ConfigProxy] info: 200 GET /api/routes
[I JupyterHub proxy:478] Adding route for Hub: / => http://127.0.0.1:8081
[ConfigProxy] info: Adding route / -> http://127.0.0.1:8081
[ConfigProxy] info: Route added / -> http://127.0.0.1:8081
[ConfigProxy] info: 201 POST /api/routes/
[I JupyterHub app:3271] JupyterHub is now running at https://:8000
[I JupyterHub log:192] 302 GET / -> /hub/ (@::ffff:XX.XXX.XXX.XX) 1.51ms
[I JupyterHub log:192] 302 GET /hub/ -> /hub/login?next=%2Fhub%2F (@::ffff:XX.XXX.XXX.XX) 0.88ms
[I JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'None:X1Y2Z3=' {'path': '/hub/', 'max_age': 3600}
[I JupyterHub log:192] 200 GET /hub/login?next=%2Fhub%2F (@::ffff:XX.XXX.XXX.XX) 111.72ms
[I JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'X1Y2Z3=:4a5b6c' {'path': '/hub/'}
[I JupyterHub base:937] User logged in: jbloggs
[I JupyterHub log:192] 302 POST /hub/login?next=%2Fhub%2F -> /hub/ (jbloggs@::ffff:XX.XXX.XXX.XX) 127.86ms
[I JupyterHub log:192] 302 GET /hub/ -> /hub/spawn (jbloggs@::ffff:XX.XXX.XXX.XX) 50.05ms
[I JupyterHub provider:660] Creating oauth client jupyterhub-user-jbloggs
[I JupyterHub spawner:1692] Spawning cylc hubapp
Failed to set groups [Errno 1] Operation not permitted
[I JupyterHub log:192] 302 GET /hub/spawn -> /hub/spawn-pending/jbloggs (jbloggs@::ffff:XX.XXX.XXX.XX) 1008.57ms
[I JupyterHub pages:399] jbloggs is pending spawn
[I JupyterHub _xsrf_utils:125] Setting new xsrf cookie for b'A1B2C3:4a5b6c' {'path': '/hub/'}
[I JupyterHub log:192] 200 GET /hub/spawn-pending/jbloggs (jbloggs@::ffff:XX.XXX.XXX.XX) 18.16ms
[I CylcHubApp mixins:541] Starting jupyterhub single-user server version 4.1.5
[I CylcHubApp mixins:555] Extending cylc.uiserver.hubapp.CylcHubApp from cylc
[I CylcHubApp mixins:555] Extending jupyter_server.serverapp.ServerApp from jupyter_server 2.14.0
[I CylcHubApp utils:73] Extension package jupyter_lsp took 0.1015s to import
[I CylcHubApp utils:73] Extension package jupyterlab took 0.4126s to import
[I CylcHubApp manager:348] cylc.uiserver | extension was successfully linked.
[I CylcHubApp manager:348] jupyter_lsp | extension was successfully linked.
[I CylcHubApp manager:348] jupyter_server_terminals | extension was successfully linked.
[I CylcHubApp manager:348] jupyterlab | extension was successfully linked.
[I CylcHubApp manager:348] notebook_shim | extension was successfully linked.
[W CylcHubApp serverapp:2107] Customizing authentication via ServerApp.login_handler_class=<class 'jupyterhub.singleuser.mixins.make_singleuser_app.<locals>.JupyterHubLoginHandler'> is deprecated in Jupyter Server 2.0. Use ServerApp.identity_provider_class. Falling back on legacy authentication.
~/jbloggs/.conda/envs/hubtest/lib/python3.9/site-packages/jupyter_server/serverapp.py:2236: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/jbloggs/logout
self.web_app = ServerWebApplication(
[I CylcHubApp manager:368] notebook_shim | extension was successfully loaded.
[I CylcHubApp manager:368] cylc.uiserver | extension was successfully loaded.
[I CylcHubApp manager:368] jupyter_lsp | extension was successfully loaded.
[I CylcHubApp manager:368] jupyter_server_terminals | extension was successfully loaded.
[I LabApp] JupyterLab extension loaded from ~/jbloggs/.conda/envs/hubtest/lib/python3.9/site-packages/jupyterlab
[I LabApp] JupyterLab application directory is ~/jbloggs/.conda/envs/hubtest/share/jupyter/lab
[I LabApp] Extension Manager is 'pypi'.
[I CylcHubApp manager:368] jupyterlab | extension was successfully loaded.
[I CylcHubApp mixins:629] Starting jupyterhub-singleuser server version 4.1.5
[I JupyterHub log:192] 200 GET /hub/api (@127.0.0.1) 0.81ms
[I CylcHubApp serverapp:3004] Serving notebooks from local directory: ~/jbloggs
[I CylcHubApp serverapp:3004] Jupyter Server 2.14.0 is running at:
[I CylcHubApp serverapp:3004] http://127.0.0.1:35241/user/jbloggs/cylc?token=123xyz
[I CylcHubApp serverapp:3004] http://127.0.0.1:35241/user/jbloggs/cylc?token=123xyz
[I CylcHubApp serverapp:3005] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C CylcHubApp serverapp:3067]
To access the server, open this file in a browser:
file://~/jbloggs/.local/share/jupyter/runtime/jpserver-93609-open.html
Or copy and paste one of these URLs:
http://127.0.0.1:35241/user/jbloggs/cylc?token=123xyz
http://127.0.0.1:35241/user/jbloggs/cylc?token=123xyz
[I CylcHubApp mixins:523] Updating Hub with activity every 300 seconds
[I CylcHubApp log:192] 302 GET /user/jbloggs/ -> /user/jbloggs/cylc? (@127.0.0.1) 2.71ms
[I JupyterHub base:1090] User jbloggs took 7.703 seconds to start
[I JupyterHub proxy:331] Adding user jbloggs to proxy /user/jbloggs/ => http://127.0.0.1:35241
[ConfigProxy] info: Adding route /user/jbloggs -> http://127.0.0.1:35241
[ConfigProxy] info: Route added /user/jbloggs -> http://127.0.0.1:35241
[ConfigProxy] info: 201 POST /api/routes/user/jbloggs
[I JupyterHub users:776] Server jbloggs is ready
[I JupyterHub log:192] 200 GET /hub/api/users/jbloggs/server/progress?_xsrf=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 6503.35ms
[I JupyterHub log:192] 200 POST /hub/api/users/jbloggs/activity (jbloggs@127.0.0.1) 76.35ms
[I JupyterHub log:192] 302 GET /hub/spawn-pending/jbloggs -> /user/jbloggs/ (jbloggs@::ffff:XX.XXX.XXX.XX) 6.34ms
[I CylcHubApp log:192] 302 GET /user/jbloggs/ -> /user/jbloggs/cylc? (@::ffff:XX.XXX.XXX.XX) 2.47ms
[I CylcHubApp log:192] 301 GET /user/jbloggs/cylc? -> cylc/ (@::ffff:XX.XXX.XXX.XX) 3.83ms
[I CylcHubApp log:192] 302 GET /user/jbloggs/cylc/ -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 14.20ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] -> /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 130.46ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (jbloggs@127.0.0.1) 112.56ms
[I JupyterHub log:192] 200 GET /hub/api/user (jbloggs@127.0.0.1) 41.16ms
[I CylcHubApp auth:1510] Logged-in user {'name': 'jbloggs', 'admin': False, 'groups': [], 'kind': 'user', 'session_id': 'A1B2C3', 'scopes': ['access:servers!server=jbloggs/', 'read:users:groups!user=jbloggs', 'read:users:name!user=jbloggs']}
[I CylcHubApp _xsrf_utils:125] Setting new xsrf cookie for b'A1B2C3:123abc' {'path': '/user/jbloggs/'}
[I CylcHubApp log:192] 302 GET /user/jbloggs/oauth_callback?code=[secret]&state=[secret] -> /user/jbloggs/cylc/ (@::ffff:XX.XXX.XXX.XX) 163.60ms
[I CylcHubApp log:192] 302 GET /user/jbloggs/cylc/assets/index-BRxHbm_z.css -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.54ms
[I CylcHubApp log:192] 302 GET /user/jbloggs/cylc/assets/index-Cs-VwyI1.js -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 2.02ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] -> /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 122.37ms
[W CylcHubApp auth:1494] oauth state argument 'e1f2g3h4' != cookie jupyterhub-user-jbloggs-oauth-state='e567fgh'
[W CylcHubApp web:1873] 403 GET /user/jbloggs/oauth_callback?code=E789HIJ&state=e1f2g3h4 (::ffff:XX.XXX.XXX.XX): oauth state does not match. Try logging in again.
[W CylcHubApp log:192] 403 GET /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (@::ffff:XX.XXX.XXX.XX) 44.85ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-jbloggs&redirect_uri=%2Fuser%2Fjbloggs%2Foauth_callback&response_type=code&state=[secret] -> /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (jbloggs@::ffff:XX.XXX.XXX.XX) 127.54ms
[W CylcHubApp auth:1494] oauth state argument 'e567fgh' != cookie jupyterhub-user-jbloggs-oauth-state=None
[W CylcHubApp web:1873] 403 GET /user/jbloggs/oauth_callback?code=F4H5IK6&state=e567fgh (::ffff:XX.XXX.XXX.XX): oauth state does not match. Try logging in again.
[W CylcHubApp log:192] 403 GET /user/jbloggs/oauth_callback?code=[secret]&state=[secret] (@::ffff:XX.XXX.XXX.XX) 2.50ms
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋
Welcome to the Jupyter community! 🎉
Update: I added some print statements in here:
jupyterhub/jupyterhub/_xsrf_utils.py
Lines 203 to 204 in b405361
and found that this function is now called in 4.1.5 on the GET requests for the JS and CSS static files, when it wasn't in 4.1.4
(Note to self: only difference between the two versions is this diff f395acd from #4771)
Further update: When inspecting the requests that were getting redirected and ultimately failing, they included the XSRF token in the Cookie
header, but didn't include the X-Xsrftoken
header, so check_xsrf_cookie()
was raising a (swallowed) 403
If I make this edit then everything works as expected:
Hidden
def check_xsrf_cookie(handler):
"""Check that xsrf cookie matches xsrf token in request"""
# overrides tornado's implementation
# because we changed what a correct value should be in xsrf_token
if not _needs_check_xsrf(handler):
# don't require XSRF for regular page views
return
token = (
handler.get_argument("_xsrf", None)
or handler.request.headers.get("X-Xsrftoken")
or handler.request.headers.get("X-Csrftoken")
+ or handler.get_cookie("_xsrf")
)
However I suspect this is defeating the point of check_xsrf_cookie()
?
I am having a very similar issue where check_xsrf_cookie returns a 403 during login.
Our cookie header contains the correct _xsrf token but the X-Xsrftoken/X-Csrftoken header is not set so the check_xsrf_cookie method returns the 403 '_xsrf' argument missing from POST.
However I suspect this is defeating the point of check_xsrf_cookie()?
Yes, that is precisely defeating the purpose. The goal of the XSRF token is establishing that the sender has access to the xsrf cookie. The XSRF token must always be sent in two places:
- the
_xsrf
cookie, which is controlled by the browser, and - another location (such as the
_xsrf
argument or X-Xsrftoken header), controlled by the requesting code
The XSRF check passes if these two match. The presence of the _xsrf
cookie itself is not meaningful on its own.
found that this function is now called in 4.1.5 on the GET requests for the JS and CSS static files, when it wasn't in 4.1.4
This generally shouldn't happen on static files, and doesn't in my tests. I will try to investigate why yours are getting caught up in this.
@ibh1127 it sounds like your login form doesn't set the xsrf token input. The explanation is above, that presence in _xsrf
is not sufficient, it must be sent by your login form as well, e.g. via hidden input, as done here.
@MetRonnie can you explain why you have added crossorigin
to your js/css? I believe this is what's causing it to take a different path, because it appears to be explciitly attempting a crossorigin-style request without the required crossorigin credentials, when a 'standard' style/script tag would work fine.
Many thanks for looking into this.
can you explain why you have added
crossorigin
to your js/css?
This seems to be done by Vite when building the web app. Removing them from the built HTML file does not solve the problem unfortunately.
However, a colleague suggested we could remove Tornado's @web.authenticated
from our static handler's get
method in the server: https://github.com/cylc/cylc-uiserver/blob/b007b5f91491a2be97fa16313dfbbbe57ce7e933/cylc/uiserver/handlers.py#L187-L188
and this does allow the JS/CSS to be fetched without a problem.
But unfortunately we still get the redirect loop problem for a XHR GET to our userprofile.json
endpoint, just without the "oauth state does not match" bit:
Log
[I JupyterHub users:776] Server rdutta is ready
[I JupyterHub log:192] 200 GET /hub/api/users/rdutta/server/progress?_xsrf=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 6968.92ms
[I JupyterHub log:192] 302 GET /hub/spawn-pending/rdutta -> /user/rdutta/ (rdutta@::ffff:XX.XXX.XXX.XX) 11.80ms
[I CylcHubApp log:192] 302 GET /user/rdutta/ -> /user/rdutta/cylc? (@::ffff:XX.XXX.XXX.XX) 0.70ms
[I CylcHubApp log:192] 301 GET /user/rdutta/cylc? -> cylc/ (@::ffff:XX.XXX.XXX.XX) 0.58ms
[W CylcHubApp web:1873] 403 POST /user/rdutta/cylc/graphql (::ffff:XX.XXX.XXX.XX): XSRF cookie does not match POST argument
[W CylcHubApp log:192] 403 POST /user/rdutta/cylc/graphql (@::ffff:XX.XXX.XXX.XX) 22.77ms
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:O63Zfgd=' {'path': '/user/rdutta/', 'max_age': 3600}
[W CylcHubApp log:192] 403 GET /user/rdutta/cylc/subscriptions (@::ffff:XX.XXX.XXX.XX) 3.63ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 5.29ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 94.55ms
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:O63Zfgd=' {'path': '/user/rdutta/', 'max_age': 3600}
[W CylcHubApp log:192] 403 GET /user/rdutta/cylc/subscriptions (@::ffff:XX.XXX.XXX.XX) 2.31ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 98.51ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 17.42ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:64a3bbe' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 123.97ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.64ms
[W CylcHubApp _xsrf_utils:198] Skipping XSRF check for insecure request GET /user/rdutta/cylc/subscriptions
[I CylcHubApp log:192] 101 GET /user/rdutta/cylc/subscriptions (rdutta@::ffff:XX.XXX.XXX.XX) 1.72ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 120.60ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 55.49ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 16.81ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:c5c009d' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 79.93ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.83ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 114.53ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 62.33ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 19.52ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:3ebc13e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 94.40ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.82ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 80.20ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 77.35ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 14.40ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:ecb92d0' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 106.67ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 10.07ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 48.52ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 67.26ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 22.32ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:577781e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 99.30ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 1.80ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 90.53ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 107.84ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 58.80ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:2a4b09d' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 2104.00ms
[I CylcHubApp log:192] 200 POST /user/rdutta/cylc/graphql (rdutta@::ffff:XX.XXX.XXX.XX) 2029.59ms
[I CylcHubApp log:192] 302 GET /user/rdutta/cylc/userprofile -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] (@::ffff:XX.XXX.XXX.XX) 11.25ms
[I JupyterHub log:192] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-rdutta&redirect_uri=%2Fuser%2Frdutta%2Foauth_callback&response_type=code&state=[secret] -> /user/rdutta/oauth_callback?code=[secret]&state=[secret] (rdutta@::ffff:XX.XXX.XXX.XX) 98.74ms
[I JupyterHub log:192] 200 POST /hub/api/oauth2/token (rdutta@127.0.0.1) 81.43ms
[I JupyterHub log:192] 200 GET /hub/api/user (rdutta@127.0.0.1) 17.16ms
[I CylcHubApp auth:1510] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'rdutta', 'session_id': '0ff7b4bd', 'scopes': ['access:servers!server=rdutta/', 'read:users:groups!user=rdutta', 'read:users:name!user=rdutta']}
[I CylcHubApp _xsrf_utils:128] Setting new xsrf cookie for b'0ff7b4bd:344d52e' {'path': '/user/rdutta/'}
[I CylcHubApp log:192] 302 GET /user/rdutta/oauth_callback?code=[secret]&state=[secret] -> /user/rdutta/cylc/userprofile (@::ffff:XX.XXX.XXX.XX) 116.27ms
Edit: just noticed on page refresh after this the app loads successfully. But after closing the browser and re-opening, I get the problem again. Seems like the XRSF cookie has not been set by the time the the GET happens. Sometimes the XSRF cookie has been set and the header is included in the GET but it still gets in a redirect loop.
One thing that would clean up and simplify things is if we only did the login redirect on Sec-Fetch-Mode: navigate
+ Sec-Fetch-Dest: document
requests. I don't think other requests are going to complete the login redirect process, so they shouldn't try. The concurrent login redirects is what's causing the oauth state messages, but that's a misdirection from the real issue, which is that the initial requests aren't accepted - what exactly happens after the failed request is less relevant.
The underlying cause is that 4.1 applies consistent XSRF checks to authenticated GET requests, which is required to protect user servers from each other, whereas 4.0 did not strictly protect GET requests, only POST and others.
In general, these are the changes required to work in this situation:
- disable XSRF checks on handlers where you don't care to protect the endpoint from other JupyterHub users:
def check_xsrf_cookie(self): return
This makes sense e.g. for shared static assets, but probably not userprofile
(removing @web.authenticated
would also avoid the xsrf check), and
2. make API requests with the JupyterHub API token in the Authorization header. Token-authenticated requests do not have XSRF checks applied. XSRF checks only apply to requests that rely on implicit auth with cookies (the source of the XSRF problem in general), OR
3. add xsrf header to GET API requests like this one for userprofile, which you already have for the graphql post, since it's been required there for longer.
I would recommend going with token approach, if possible, but copying the xsrf header code you already have to the requests missing them is probably the smallest change to get things working in the short term.
I've made the following PRs:
- cylc/cylc-uiserver#592 (fixes static asset requests)
- cylc/cylc-ui#1778 (fixes userprofile request by adding xsrf token)
which together made cylc work for me in 4.1.5 and should be fully backward compatible.
Much appreciated! And thank you for the explanation!
It now works after logging in for the first time. However, if I close the browser, then re-open the browser, I find that I get the oauth redirect loop again. I think this is because the XSRF cookie has session lifetime, and when re-opening the browser it is no-longer set in time for the userprofile GET request.
I think this is why the redirect loop is happening: After the first redirect to the oauth URLs, the XSRF cookie is set, but when it redirects back to the userprofile, the GET request still does not contain the XSRF token header (because it is still the original request? (It is not re-executing our axios.get()
)). Hence it redirects back to oauth and so on.
This can be worked around by retrying the userprofile GET after the redirect loop fails. But it sounds like this could be solved by using the JHub API token that you've mentioned, is there somewhere in the docs you can point me to for this? Or is there another way to solve this by somehow going through the oauth before the userprofile GET request?
Ah, the missing xsrf token is probably because the request to serve index.html
doesn't set the xsrf token because that page is also served by the StaticFileHandler (in most Jupyter pages, HTML is served by a template, and the xsrf token is computed for the template namespace). I believe that is fixed by the latest commit in cylc/cylc-uiserver#592, which accesses the xsrf_token for any static page GET.
That way, when GET /user/:name/cylc/
completes, you definitely have a currently valid xsrf cookie
That's almost got it, however I think when the index.html
is cached by the browser, the GET never goes through our static handler and so the XSRF cookie is not set 😢. This seems more prevalent on Firefox than Chrome
Nothing that can't be solved by refreshing the page, however.
Edit: see cylc/cylc-uiserver#592 (comment):
I think setting
Cache-Control: no-cache
does the trick 🤞
Awesome, thanks for the fixes!