Respect kernel from Jupyter/Jupytext kernelspec when specified

[Thomas Vandal]

Is your feature request related to a problem? Please describe.
When an ipynb is converted with Jupytext, the kernelspec is kept in the header (see below for example). That way, if a kernel is set (for e.g. manualy in the jupytext header, or with a tool like Juypterlab), the info is kept in the Python file. It would be nice if:

1. When there is a jupytext header, there kernel could be automatically selected based on the kernelspec -> name info.
2. There was a similar feature (header with metadata) in Jupynium files.

Describe alternatives you've considered
For now, I'm using a mapping to :JupyniumKernelSelect, which works well. The main advantage of reading metadata would be to avoid re-setting the kernel time I open the notebook.

Example Jupytext header:

# ---
# jupyter:
#   jupytext:
#     cell_metadata_filter: -all
#     formats: ipynb,py:percent
#     text_representation:
#       extension: .py
#       format_name: percent
#       format_version: '1.3'
#       jupytext_version: 1.14.5
#   kernelspec:
#     display_name: testjup
#     language: python
#     name: testjup
# ---

I think this is the type of thing I would be able to work on at some point and submit a PR, if you want. It mostly means reading extra info from an input file, and the Jupyter API commands should be the same as when selecting a kernel manually, right?

Thank you!

[Kiyoon Kim]

This only applies for creating a new ipynb file with :JupyniumStartSync right?

Currently, it will detect the conda environment you're running with CONDA_PREFIX environment variable and will choose the right kernel for you. Does this workflow not work for you? you'd probably want to activate an environment anyway (for LSPs to work properly) so I thought detecting the current environment makes more sense than parsing the header from Jupytext.

[Thomas Vandal]

This only applies for creating a new ipynb file with :JupyniumStartSync right?

I think so, yes.

Currently, it will detect the conda environment you're running with CONDA_PREFIX environment variable and will choose the right kernel for you. Does this workflow not work for you? you'd probably want to activate an environment anyway (for LSPs to work properly) so I thought detecting the current environment makes more sense than parsing the header from Jupytext.

I don't use conda, but adding support for VIRTUAL_ENV would indeed fix my issue and make more sense than parsing header from Jupytext. However, for venv there might be some ambiguity on how to match the environment, as there is no conda_env_path in the metadata.

My typical workflow is:

• Create virtual environment with venv
• Register the environment with python -m ipykernel install --user --name=<envname>
• Most of the time, <envname> is what comes directly before venv in the env path (e.g. ~/repos/somerepo/venv would have name somerepo. However, for other users with a similar workflow, the name might be different...

I would see 2 ways of getting the proper environment match for such a case:

1. os.path.basename(os.path.split(conda_env_path)[-2]) == kernel_specs["testjup"]["spec"]["display_name"] (but someone might register the environment with ipykernel to a completely different name from the parent directory, in which case this fails)
2. kernel_specs["testjup"]["spec"]["argv"][0] == os.path.join(conda_env_path, "bin", kernel_specs["testjup"]["spec"]["language"])

2 seems like the most robust option to me. Also, I kept conda_env_path in my test code but using a distinct name/mechanism for virtual envs might be best.

I can send a PR implementing something along those lines (or another option if you have a better idea for how to tackle this).

Thanks!

[Kiyoon Kim]

I haven't used venv for notebook so I'll need to see how it works. For conda, you just have to install conda install nb_conda_kernels in the base environment and it will register all environments automatically.

In your solutions, I don't understand why there's conda_env_path? I thought the solution is for virtual environment?

[Thomas Vandal]

Oh sorry I forgot to mention, the code lines above are from tests I did where local conda_env_path = vim.env.VIRTUAL_ENV, instead of CONDA_PREFIX (I kept the variable name the same everywhere in the code to do a quick test).

[Kiyoon Kim]

I like the solution 2. Feel free to open a PR, and please add a brief explanation in comments of the detection logic. Also, make sure it will not fail on KeyError and IndexError. If some environment doesn't have such kernelspec just ignore it and move on, defaulting to no action if no matches found. Thank you!