jupyter / docker-stacks

What docker image(s) are you using?

datascience-notebook

Host OS system

Ubuntu 20.04

Host architecture

x86_64

What Docker command are you running?

I don't use docker but apptainer running in a HPC cluster

It runs the image as read-only (I cannot set it read/write as that would disable GPU passthrough)

How to Reproduce the problem?

Run the container mounting your home inside the container and use the NB_USER env variable to set it to your own user

Observe that XDG_CACHE_HOME still points to /home/jovyan/.cache/ (which is a little unexpected when you have set the user)

This makes it impossible to use conda to ex. create an environment in your writeable home..because it will try to write to the cache directory from the readonly container image

Command output

No response

Expected behavior

XDG_CACHE_HOME is set correctly when NB_USER is set

Actual behavior

XDG_CACHE_HOME is always /home/jovyan/.cache/

Anything else?

I have got around it by passing --env XDG_CACHE_HOME=${HOME}/.cache myself when starting the container with apptainer

Latest Docker version

I've updated my Docker version to the latest available, and the issue persists

This should be set here (and preserved): https://github.com/jupyter/docker-stacks/blob/main/images/docker-stacks-foundation/start.sh#L125

@HasseJohansen Do you build the image yourself? Because this project does not provide GPU accelerated images.

Hi @benz0li and thank you for the swift reply

I did notice the line in the start script setting XDG_CACHE_HOME, but somehow it doesn't work for me. I need to pass something like this when running with apptainer --env ${HOME}/cache

I actually retested without passing the above env and had the XDG_CACHE_HOME set to /home/jovyan/.cache which when using apptainer is readonly by default

I will try to reproduce with docker

I am not rebuilding the image. We get it through the singularity-hpc project (https://singularityhub.github.io/singularity-hpc/) which are using official images, but adding a GitHub hosted "recipe" for each upstream repo(in this instance Jupyter docker stacks) to build an env module(https://modules.readthedocs.io/en/latest/)

It will also convert the docker image to the .sif image format used by apptainer

As apptainer is working somewhat transparently on the host

We can do something like module load jupyter/datascience-notebook and then ex. run start-notebook.sh which is a script exposed by the env module

apptainer has an option --nv to use gpu's in the container(it will do some different things I cannot fully remember right now. They do some LD_PRELOAD stuff)

BR
Hasse

Ok. I think I know the problem. Using the image with apptainer would newer run the start.sh/entrypoint script

I can reproduce with docker doing something like this(emulating some of stuff apptainer does by default)

docker run -it --rm --read-only -v /home/<user>/testhome:/home/<user> -v /etc/passwd:/etc/passwd:ro -e HOME=/home/<user> -e CHOWN_HOME=true --user 1000 -e NB_USER=<user> jupyter/datascience-notebook /bin/bash

<user>@7694277d45e4:~$ env|grep HOME
HOME=/home/<user>
CHOWN_HOME=true
XDG_CACHE_HOME=/home/jovyan/.cache/

<user>@7694277d45e4:~$ conda create -n torch
CondaError: Error encountered while attempting to create cache directory.
Directory: /home/jovyan/.cache/conda/notices     Exception: [Errno 30] Read-only file system: '/home/jovyan/.cache/conda'

As we don't use start.sh XDG_CACHE_HOME must come from the Dockerfile at build time (also notice that if you use Jupyter/minimal-notebook instead XDG_CACHE_HOME is not set which in my opinion is better as it will just use $HOME/.cache as default)

It comes from the scipy-notebook Dockerfile which is inherited by the datascience-notebook

It comes from the scipy-notebook Dockerfile which is inherited by the datascience-notebook

@mathbunnyru Why not use ARG XDG_CACHE_HOME="/home/${NB_USER}/.cache/" at

docker-stacks/images/scipy-notebook/Dockerfile

Line 78 in 278dd76

ENV XDG_CACHE_HOME="/home/${NB_USER}/.cache/"

and get rid of

docker-stacks/images/docker-stacks-foundation/start.sh

Lines 124 to 125 in 278dd76

    
           # Update potentially outdated environment variables since the image build 
        
           export XDG_CACHE_HOME="/home/${NB_USER}/.cache"

?

@mathbunnyru Does matplotlib strictly require XDG_CACHE_HOME to be set?

I have no idea, but I think we can try it. XDG_CACHE_HOME was indeed added here and only for matplotlib cache: https://github.com/jupyter/docker-stacks/pull/289/files

@mathbunnyru Does matplotlib strictly require XDG_CACHE_HOME to be set?

@benz0li could you create a PR removing two exports? One in the Dockerfile and another in start.sh.
I think if it works, we should be fine.

@benz0li could you create a PR removing two exports?

Will do.

@HasseJohansen thanks to @benz0li we removed manually-set $XDG_CACHE_HOME.
This should fix your problem.

New images will be ready in approximately 1h30minutes.
One thing worth mentioning - we push our images to Quay.io registry, so, to receive this fix you will have to use images from there.

Thanks. That was quick. I Think it will make the image easier to use in readonly environments. It at least fooled me that it wrote the cache in another users home when I hadn't told it to🙂

Thanks. That was quick.

@HasseJohansen Could you please confirm that it works as expected now?

Thank you.

(I do not use apptainer and therefore cannot test)

	# Update potentially outdated environment variables since the image build
	export XDG_CACHE_HOME="/home/${NB_USER}/.cache"

Setting NB_USER for running the container should also set XDG_CACHE_HOME to that users home