'_xsrf' argument missing from POST (jupyterhub 4.1.0, jupyter-server-proxy 4.1.2)
vivian-rook opened this issue · comments
Bug description
Upgrading from zero to jupyterhub chart version 3.2.1 to 3.3.2 Most things seem fine, but externally launching applications, namely:
https://github.com/jupyterhub/jupyter-rsession-proxy
https://github.com/innovationOUtside/nb_serverproxy_openrefine
Do not open, giving an error in the browser (for RStudio) Status code 403 returned by RStudio Server when executing 'client_init' and in the singleuser container logs:
[E 2024-03-21 19:39:48.509 ServerApp] Not accepting cookie auth on POST /user/VRook%20%28WMF%29/rstudio/rpc/client_init: HTTP 403: Forbidden ('_xsrf' argument missing from POST)
[W 2024-03-21 19:39:48.510 ServerApp] 403 POST /user/VRook%20%28WMF%29/rstudio/rpc/client_init (10.100.1.0): '_xsrf' argument missing from POST
How to reproduce
Install zero to jupyterhub chart version 3.3.2 with rstudio built into the singleuser container and launch rstudio.
Expected behaviour
RStudio/OpenRefine open
Actual behaviour
RStudio/OpenRefine give an error
Your personal set up
zero-to-jupyterhub chart 3.3.2 launching singleuser containers built out of ubuntu 22.04
@PAWS:~$ pip freeze
aiohttp==3.9.3
aiosignal==1.3.1
alembic==1.13.1
altair==5.2.0
annotated-types==0.6.0
anyio==4.3.0
argon2-cffi==23.1.0
argon2-cffi-bindings==21.2.0
arrow==1.3.0
asttokens==2.4.1
async-generator==1.10
async-lru==2.0.4
async-timeout==4.0.3
attrs==23.2.0
Babel==2.14.0
bash_kernel==0.9.3
beautifulsoup4==4.12.3
bleach==6.1.0
blinker==1.7.0
bokeh==3.3.4
bqplot==0.12.43
branca==0.7.1
cachetools==5.3.3
certifi==2024.2.2
certipy==0.1.3
cffi==1.16.0
charset-normalizer==3.3.2
cli_helpers==2.3.1
click==8.1.7
colorcet==3.1.0
comm==0.2.2
configobj==5.0.8
contourpy==1.2.0
cryptography==36.0.2
cycler==0.12.1
debugpy==1.8.1
decorator==5.1.1
defusedxml==0.7.1
deltas==0.7.0
docopt==0.6.2
exceptiongroup==1.2.0
executing==2.0.1
fastapi==0.110.0
fastjsonschema==2.19.1
Flask==3.0.2
fonttools==4.50.0
fqdn==1.5.1
frozenlist==1.4.1
gitdb==4.0.11
GitPython==3.1.42
google==3.0.0
greenlet==3.0.3
h11==0.14.0
holoviews==1.18.3
httpcore==1.0.4
httpx==0.27.0
idna==3.6
ipydatawidgets==4.3.5
ipykernel==6.29.3
ipyleaflet==0.18.2
ipython==8.22.2
ipywidgets==8.1.2
isoduration==20.11.0
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.3
joblib==1.3.2
json5==0.9.24
jsonable==0.3.1
jsonpointer==2.4
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
jupyter==1.0.0
jupyter-console==6.6.3
jupyter-events==0.10.0
jupyter-lsp==2.2.4
jupyter-resource-usage==1.0.2
jupyter-rsession-proxy==2.2.1
jupyter-telemetry==0.1.0
jupyter_client==8.6.1
jupyter_core==5.7.2
jupyter_server==2.13.0
jupyter_server_proxy==4.1.2
jupyter_server_terminals==0.5.3
jupyterhub==4.1.0
jupyterlab==4.1.5
jupyterlab-link-share==0.3.0
jupyterlab_pygments==0.3.0
jupyterlab_server==2.25.4
jupyterlab_widgets==3.0.10
kiwisolver==1.4.5
labpawspublic @ git+https://github.com/toolforge/labpawspublic@08b8b4f89143055df024ebbbfb159b07f0f2a5b0
Lasagne==0.1
linkify-it-py==2.0.3
lxml==5.1.0
Mako==1.3.2
Markdown==3.6
markdown-it-py==3.0.0
MarkupSafe==2.1.5
matplotlib==3.8.3
matplotlib-inline==0.1.6
mdit-py-plugins==0.4.0
mdurl==0.1.2
memento-client==0.6.1
mistune==3.0.2
mpmath==1.3.0
multidict==6.0.5
mwapi==0.6.1
mwcli==0.0.3
mwdb==0.1.2
mwdiffs==0.0.2
mwoauth==0.4.0
mwparserfromhell==0.6.6
mwpersistence==0.2.4
mwreverts==0.1.5
mwsessions==0.0.2
mwtypes==0.3.2
mwxml==0.3.3
mycli==1.27.0
nb-serverproxy-openrefine @ git+https://github.com/innovationOUtside/nb_serverproxy_openrefine@828089e0397defe011e43ddd1e2ba067afc43b6a
nbclient==0.10.0
nbconvert==7.16.3
nbformat==5.10.3
nest-asyncio==1.6.0
nose==1.3.7
notebook==7.1.2
notebook_shim==0.2.4
numpy==1.26.4
oauthlib==3.2.2
overrides==7.7.0
packaging==23.2
pamela==1.1.0
pandas==2.2.1
pandocfilters==1.5.1
panel==1.3.8
para==0.0.8
param==2.0.2
parso==0.8.3
paws @ git+https://github.com/toolforge/ipynb-paws@147bbeb4dda35e94785814ca9a713c044229b25d
pexpect==4.9.0
pillow==10.2.0
platformdirs==4.2.0
plotly==5.20.0
pprofile==2.1.0
prometheus_client==0.20.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
py-heat==0.0.6
py-heat-magic==0.0.2
pyaes==1.6.1
pyarrow==15.0.2
pycparser==2.21
pydantic==2.6.4
pydantic_core==2.16.3
pydeck==0.8.1b0
pydot==2.0.0
Pygments==2.17.2
PyJWT==2.8.0
PyMySQL==1.1.0
pyOpenSSL==24.1.0
pyparsing==3.1.2
pyperclip==1.8.2
python-dateutil==2.9.0.post0
python-json-logger==2.0.7
python-stdnum==1.20
python-wdqs @ git+https://github.com/yuvipanda/python-wdqs.git@272dc435bdd1778561a486b42521a87d5e80c516
pythreejs==2.4.2
pytz==2024.1
pyviz_comms==3.0.1
pywikibot @ file:///srv/paws/pwb
PyYAML==6.0.1
pyzmq==25.1.2
qtconsole==5.5.1
QtPy==2.4.1
referencing==0.34.0
requests==2.31.0
requests-oauthlib==1.4.0
rfc3339-validator==0.1.4
rfc3986-validator==0.1.1
rich==13.7.1
rise==5.7.1
rpds-py==0.18.0
rpy2==3.5.15
ruamel.yaml==0.18.6
ruamel.yaml.clib==0.2.8
scikit-learn==1.4.1.post1
scikit-neuralnetwork==0.7
scipy==1.12.0
seaborn==0.13.2
Send2Trash==1.8.2
simpervisor==1.0.0
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
soupsieve==2.5
SQLAlchemy==2.0.28
sqlglot==23.0.5
sqlparse==0.4.4
sseclient==0.0.22
stack-data==0.6.3
starlette==0.36.3
stopit==1.1.2
streamlit==1.32.2
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
terminado==0.18.1
Theano==1.0.5
threadpoolctl==3.4.0
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
toolz==0.12.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.2
traittypes==0.2.1
types-python-dateutil==2.9.0.20240316
typing_extensions==4.10.0
tzdata==2024.1
tzlocal==5.2
uc-micro-py==1.0.3
uri-template==1.3.0
urllib3==2.2.1
watchdog==4.0.0
wcwidth==0.2.13
webcolors==1.13
webencodings==0.5.1
websocket-client==1.7.0
Werkzeug==3.0.1
widgetsnbextension==4.0.10
wmpaws==1.3
xyzservices==2023.10.1
yamlconf==0.2.3
yarl==1.9.4
@minrk this would be a jupyter-rsession-proxy issue, on being compatible with 4.1.0 - whats your take?
@minrk this would be a jupyter-rsession-proxy issue, on being compatible with 4.1.0 - whats your take?
I was kind of thinking that, though it happened with two (the only two external plugins that I have) at the same time. So it seemed a little more like it belonged here, though perhaps not.
Its a change in jupyterhub (software deployed by the helm chart, and no the helm chart project itself) that has caused this i think, but its security related breaking change rather than a bug i think. So the fix could be for the software interacting with jupyterhub to adjust to jupyterhub's new behavior rather than for jupyterhub to revert the change.
I've now tried to reproduce this, but failed - the image rocker/binder:4.3.3 launched to start jupyterhub-singleuser worked as intended, and it provides RStudio.
The issue will likely boil down to details on whats going on in the user environment, for example, are you using old versions of jupyterhub inside the user environment image or similar?
@consideRatio I do not believe it is JupyterHub 4.1 changes, but rather jupyter-server-proxy changes that would be responsible. I think you may see this with current jupyter-server-proxy and ~any Jupyter server.
I don't know how jupyter-rsession-proxy works or how you can influence the authentication of the requests it makes.
Both the old and the, attempted, upgraded instances are running jupyter_server_proxy 4.1.2 So that doesn't look like it changed. I went from jupyterhub 4.0.2 to 4.1.0 on both the singleuser container and the hub-k8s container.
I agree that it is likely something about my environment if RStudio is working in another deploy. Let me see if I can get an instance of rocker/binder:4.3.3 to see if it is working in the rest of my env.
ok, then it would probably be in jupyterhub 4.1, thanks for narrowing it down!
I've confirmed this with JupyterHub 4.1 and jupyter-server-proxy 4.1.2. I'm not sure this is a situation that should have changed, so I will investigate.
@consideRatio fwiw, rocker/binder:4.3.3 has jupyter-server-proxy 4.1.1 and jupyterhub 4.0.2, so isn't up to date on either package.
okay, I see and understand what's different. I will try to figure out if I can make a JupyterHub 4.1.1 that unbreaks this.
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋
Welcome to the Jupyter community! 🎉
@minrk Thanks for #4745, it seems to have fixed my issues with jupyter-server-proxy but I'm seeing the same behaviour for some other stuff.
For example, jupyter-templates isn't able to activate properly because it's getting a 403 on it's handler.
Same for nbgallery, I'm seeing a bunch of 403s and I can't import a notebook from our nbgallery instance to my Jupyter server. (note that nbgallery depends on "ServerApp.disable_check_xsrf=True" for the notebook import feature to work).
Some bits from my sanitized logs that might be of help:
/opt/conda/lib/python3.10/site-packages/jupyter_server/serverapp.py:2235: JupyterServerAuthWarning: Core endpoints without @allow_unauthenticated, @ws_authenticated, nor @web.authenticated:
- GET of JupyterHubLogoutHandler registered for /user/$JUPYTERHUB_USER/logout
- GET of JupyterHubOAuthCallbackHandler registered for /user/$JUPYTERHUB_USER/oauth_callback
self.web_app = ServerWebApplication(
[...]
[I 2024-03-25 18:27:05.673 ServerApp] Logged-in user {'kind': 'user', 'admin': True, 'name': '$JUPYTERHUB_USER', 'groups': [], 'session_id': 'some-session-id', 'scopes': ['access:servers!server=$JUPYTERHUB_USER/', 'read:users:groups!user=$JUPYTERHUB_USER', 'read:users:name!user=$JUPYTERHUB_USER']}
[E 2024-03-25 18:27:05.674 ServerApp] Not accepting cookie auth on GET /user/$JUPYTERHUB_USER/oauth_callback: HTTP 403: Forbidden ('_xsrf' argument missing from GET)
[I 2024-03-25 18:27:05.674 ServerApp] 302 GET /user/$JUPYTERHUB_USER/oauth_callback?code=[secret]&state=[secret] -> /user/$JUPYTERHUB_USER/templates/names (@127.0.0.1) 57.94ms
[...]
[W 2024-03-25 18:27:08.925 ServerApp] 403 GET /user/$JUPYTERHUB_USER/jupyterlab_nbgallery/environment?_=1711391214409 (@127.0.0.1) 2.76ms
[E 2024-03-25 18:27:08.925 ServerApp] Not accepting cookie auth on GET /user/$JUPYTERHUB_USER/jupyterlab_nbgallery/instrumentation: HTTP 403: Forbidden ('_xsrf' argument missing from GET)
[...]
[E 2024-03-25 18:48:44.067 ServerApp] Not accepting cookie auth on POST /user/$JUPYTERHUB_USER/post/example.ipynb: HTTP 403: Forbidden ('_xsrf' argument missing from POST)
[W 2024-03-25 18:48:44.067 ServerApp] 403 POST /user/$JUPYTERHUB_USER/post/example.ipynb (127.0.0.1): '_xsrf' argument missing from POST
[W 2024-03-25 18:48:44.068 ServerApp] wrote error: "'_xsrf' argument missing from POST"
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/tornado/web.py", line 1769, in _execute
result = await result # type: ignore
File "/opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 750, in prepare
await super().prepare()
File "/opt/conda/lib/python3.10/site-packages/jupyter_server/base/handlers.py", line 632, in prepare
self.check_xsrf_cookie()
File "/opt/conda/lib/python3.10/site-packages/jupyterhub/services/auth.py", line 915, in <lambda>
cls.check_xsrf_cookie = lambda handler: self.check_xsrf_cookie(handler)
File "/opt/conda/lib/python3.10/site-packages/jupyterhub/services/auth.py", line 924, in check_xsrf_cookie
check_xsrf_cookie(handler)
File "/opt/conda/lib/python3.10/site-packages/jupyterhub/_xsrf_utils.py", line 181, in check_xsrf_cookie
raise web.HTTPError(
tornado.web.HTTPError: HTTP 403: Forbidden ('_xsrf' argument missing from POST)
[W 2024-03-25 18:48:44.069 ServerApp] 403 POST /user/$JUPYTERHUB_USER/post/example.ipynb (@127.0.0.1) 2.48ms
To be fair, I jumped from JupyterHub 3.0.0 to now 4.1.1 with my chart update so I might be having other issues. I can create a new GitHub issue if you would like but I thought that this might all be related.
@cccs-nik definitely related, sorry for the trouble!
The updated implementation ignores the Jupyter-server setting that disables xsrf checks, which I had forgotten about. Of course, disbling xsrf checks also by definition means enabling xsrf attacks, but the setting is there so it should be respected for folks who know what they are doing.
I'd argue you should probably be using token-authenticated requests if you can, and I'd be happy to help advise on that, and would also like to hear why it doesn't work for you. XSRF checks don't apply to Token-authenticated requests, so none of this comes up.
4.1.2 does allow disabling xsrf on a per-Handler basis, though, if you override check_xsrf_cookie you can make it a no-op (this is the same as before, but unbroken in 4.1.1).
Can you open a new Issue on JupyterHub?
@minrk Thanks for the quick answer! I opened #4752. My JupyterHub instance runs in a heavily restricted environment (practically cut-off from the internet) so I'm not too worried about disabling xsrf checks but I'm always happy to improve our security and deployment.
Would using token-authenticated requests require modifications to nbgallery and jupyterlab_templates or is that something I can tweak in my Helm chart? To be clear, I don't own or maintain either extensions. We already have a custom OAuth authenticator which might be of help for what you're suggesting.
Disabling xsrf for the problematic handlers I understand how to do, but that's for sure a contribution to the extensions unless I'm not thinking of an alternative approach.
Would using token-authenticated requests require modifications to nbgallery and jupyterlab_templates
I believe it requires updating the extensions: nbgallery/lab-extensions#58