Conda environment not fully set in Jupyter

[Nicolas M. Thiéry]

What docker image(s) are you using?

minimal-notebook

Host OS system

CentOS

Host architecture

x86_64

What Docker command are you running?

sudo docker run -p8888:8888 --rm docker.io/jupyter/minimal-notebook:latest

(or standard run from JupyterHub)

How to Reproduce the problem?

Start a Python or C++ kernel, e.g. from a notebook and run the following
command:

!env | grep CONDA 

Command output

CONDA_DIR=/opt/conda

Expected behavior

See all the usual environment variables that are set when a conda
environment is set; e.g.:

CONDA_EXE=/opt/conda/bin/conda
CONDA_PREFIX=/opt/conda
CONDA_PROMPT_MODIFIER=(base) 
_CE_CONDA=
CONDA_SHLVL=1
CONDA_DIR=/opt/conda
CONDA_PYTHON_EXE=/opt/conda/bin/python
CONDA_DEFAULT_ENV=base

Actual behavior

If a barebone shell is launched from Jupyter (e.g. through shebang calls in a notebook), the conda environment variables are not set.

Anything else?

Suggestion

Launch Jupyter itself within an activated conda environment. Maybe by using an entry
point such as conda run jupyter ....

For now I am using as workaround:

%bash --login
...

but this is tricky to find for end users.

Use case

In a C++ course, we include calls to compilation commands in the course
narrative to explain how to compile; students also include them
in the support of their presentation to run the compilation and execution
of their programs. These compilation commands often need a fully
configured conda environment.

Latest Docker version

• I've updated my Docker version to the latest available, and the issue persists

[Ayaz Salikhov]

This was the solution which was implemented the last time:
https://github.com/jupyter/docker-stacks/pull/1166/files

echo 'eval "$(command conda shell.bash hook 2> /dev/null)"' >> /etc/skel/.bashrc
This command is executed before creating NB_USER, meaning it's .bashrc will have this line in the shell.
But it only works for terminals, not when opening Jupyter Notebook (that's why the current issue is only present in Jupyter Notebooks but not the terminals)

I'm currently thinking about a different solution:
Create a file with a similar line in the /usr/local/bin/before-notebook.d/ directory.
This will source the file right before running the jupyter command, so it should work for all the subprocesses.
There is one downside, in case --user root, this will run under root, but it mostly sets env variables, so we should be fine.

Another approach is to manually set all the variables in the ENV docker command (not just CONDA_DIR, which is why it works now).

@benz0li any thoughts here?

[Olivier Benz]

@benz0li any thoughts here?

@mathbunnyru I use a systemwide IPython configuration for things like that.

E.g. https://github.com/b-data/jupyterlab-python-docker-stack/blob/2cd13000a7582c5ff1cb6aa2b3c930877432ed14/base/conf/ipython/usr/local/etc/ipython/ipython_config.py#L3-L24

import os

if (
    os.path.exists(os.path.join(os.environ["HOME"], "bin"))
    and not os.path.join(os.environ["HOME"], "bin") in os.getenv("PATH", "")
):
    os.environ["PATH"] = (
        os.path.join(os.environ["HOME"], "bin")
        + os.pathsep
        + os.getenv("PATH", "")
    )

if (
    os.path.exists(os.path.join(os.environ["HOME"], ".local", "bin"))
    and not os.path.join(os.environ["HOME"], ".local", "bin")
    in os.getenv("PATH", "")
):
    os.environ["PATH"] = (
        os.path.join(os.environ["HOME"], ".local", "bin")
        + os.pathsep
        + os.getenv("PATH", "")
    )

to mimick

if [ -d "$HOME/bin" ] && [[ "$PATH" != *"$HOME/bin"* ]] ; then
    PATH="$HOME/bin:$PATH"
fi

if [ -d "$HOME/.local/bin" ] && [[ "$PATH" != *"$HOME/.local/bin"* ]] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

in ~/.zshrc.

[Olivier Benz]

I'm currently thinking about a different solution: Create a file with a similar line in the /usr/local/bin/before-notebook.d/ directory. This will source the file right before running the jupyter command, so it should work for all the subprocesses.

This should work and seems a better solution to resolve the issue.

There is one downside, in case --user root, this will run under root, but it mostly sets env variables, so we should be fine.

True.

[Olivier Benz]

https://docs.conda.io/projects/conda/en/4.14.x/dev-guide/deep-dive-activation.html

[Ayaz Salikhov]

Thanks! I see you

@benz0li any thoughts here?

@mathbunnyru I use a systemwide IPython configuration for things like that.

E.g. https://github.com/b-data/jupyterlab-python-docker-stack/blob/2cd13000a7582c5ff1cb6aa2b3c930877432ed14/base/conf/ipython/usr/local/etc/ipython/ipython_config.py#L3-L24

import os

if (
    os.path.exists(os.path.join(os.environ["HOME"], "bin"))
    and not os.path.join(os.environ["HOME"], "bin") in os.getenv("PATH", "")
):
    os.environ["PATH"] = (
        os.path.join(os.environ["HOME"], "bin")
        + os.pathsep
        + os.getenv("PATH", "")
    )

if (
    os.path.exists(os.path.join(os.environ["HOME"], ".local", "bin"))
    and not os.path.join(os.environ["HOME"], ".local", "bin")
    in os.getenv("PATH", "")
):
    os.environ["PATH"] = (
        os.path.join(os.environ["HOME"], ".local", "bin")
        + os.pathsep
        + os.getenv("PATH", "")
    )

to mimick

if [ -d "$HOME/bin" ] && [[ "$PATH" != *"$HOME/bin"* ]] ; then
    PATH="$HOME/bin:$PATH"
fi

if [ -d "$HOME/.local/bin" ] && [[ "$PATH" != *"$HOME/.local/bin"* ]] ; then
    PATH="$HOME/.local/bin:$PATH"
fi

in ~/.zshrc.

Thank you. I understand your solution. I think I would like to use the DRY principle and try to have conda activation in only one place.

I'm currently thinking about a different solution: Create a file with a similar line in the /usr/local/bin/before-notebook.d/ directory. This will source the file right before running the jupyter command, so it should work for all the subprocesses.

This should work and seems a better solution to resolve the issue.

There is one downside, in case --user root, this will run under root, but it mostly sets env variables, so we should be fine.

True.

👍

[Ayaz Salikhov]

@nthiery I merged the fix.
New images will be ready in approximately 1 hour.