Unable to attach nvim to server on Windows

Question

Unable to attach nvim to server on Windows

sho-87 opened this issue a year ago · comments

I'm following the usage instructions and when I run :JupyniumStartAndAttachToServer in nvim, a window opens for a few seconds and closes - no browser or notebook appears

pip freeze shows that both jupynium and selenium packages are installed:

jupynium @ file:///C:/Users/Simon/AppData/Local/nvim-data/lazy/jupynium.nvim
selenium==4.8.0

Testing selenium from python also works OK:

from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://www.selenium.dev/selenium/web/web-form.html")

When I simply run jupynium from the terminal, firefox opens showing the notebook tree and the following info:

jupynium.cmds.jupynium:  439 - INFO - No nvim attached. Waiting for nvim to attach. Run jupynium --nvim_listen_addr /tmp/example (use `:echo v:servername` of nvim)

At this point, I go back to nvim and run :JupyniumAttachToServer but I get the following exception:

jupynium.cmds.jupynium:  225 - INFO - New nvim wants to attach: Namespace(attach_only=True, check_running=False, firefox_profile_name=None, firefox_profiles_ini_path=None, jupyter_command=['jupyter'], notebook_URL='http://localhost:8888/tree/', notebook_dir=None, nvim_listen_addr='\\\\\\\\.\\\\pipe\\\\nvim.24356.0', sleep_time_idle=0.05, version=False)
jupynium.pynvim_helpers:   11 - INFO - nvim addr: \\\\.\\pipe\\nvim.24356.0
jupynium.cmds.jupynium:  249 - ERROR - Exception occurred while attaching a new nvim. Ignoring.
Traceback (most recent call last):
  File "C:\Users\Simon\AppData\Roaming\Python\Python38\site-packages\jupynium\cmds\jupynium.py", line 230, in attach_new_neovim
    nvim = attach_and_init(new_args.nvim_listen_addr)
  File "C:\Users\Simon\AppData\Roaming\Python\Python38\site-packages\jupynium\pynvim_helpers.py", line 28, in attach_and_init
    raise TimeoutError("Timeout while waiting for nvim to start")
TimeoutError: Timeout while waiting for nvim to start

Not sure if this is related, but nvim doesn't seem to have access to any of the sync commands like JupyniumStartSync. it only sees these:

Kiyoon Kim · Answer 1 · Tue Feb 14 2023 08:41:24 GMT+0800 (China Standard Time)

First of all, the commands will be initialised when you attach to the server, so that's normal.

I see that you're using Windows. Let me try on Windows and I'll get back to you soon.

Kiyoon Kim · Answer 2 · Tue Feb 14 2023 10:30:35 GMT+0800 (China Standard Time)

What is your:echo &shell? Is it PowerShell?

Simon Ho · Answer 3 · Tue Feb 14 2023 11:38:21 GMT+0800 (China Standard Time)

im launching nvim in powershell yes, but when I run :echo &shell it says cmd.exe, which is strange

I also use neovide...that says cmd.exe too

Kiyoon Kim · Answer 4 · Tue Feb 14 2023 11:50:29 GMT+0800 (China Standard Time)

im launching nvim in powershell yes, but when I run :echo &shell it says cmd.exe, which is strange

I also use neovide...that says cmd.exe too

That is not strange. Unless you explicitly set it to powershell it will be cmd by default. It matters only when you execute shell commands inside vim.

I think many plugins assume cmd, so you may as well leave the setting as is. I'm still going to make this plugin compatible with powershell anyway though.

Can you test with branch fix/windows?

Kiyoon Kim · Answer 5 · Tue Feb 14 2023 11:54:30 GMT+0800 (China Standard Time)

On my end the branch works, but I still have one last problem. I can't close neovim when I attach it to the server once.

Simon Ho · Answer 6 · Tue Feb 14 2023 12:07:34 GMT+0800 (China Standard Time)

JupyniumStartAndAttachToServer works now - it starts notebook in firefox and asks me for password

2 problems at this stage:

the notebook start directory is always ~, even if I start nvim from a different directory (e.g. > cd Desktop > nvim), or if I change the cwd from within nvim
when I run JupyniumStartSync I get a kernel error:

Traceback (most recent call last):
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\web.py", line 1704, in _execute
    result = await result
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\Simon\anaconda3\lib\site-packages\notebook\services\sessions\handlers.py", line 69, in post
    model = yield maybe_future(
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\Simon\anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 98, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "C:\Users\Simon\anaconda3\lib\site-packages\notebook\services\sessions\sessionmanager.py", line 110, in start_kernel_for_session
    kernel_id = yield maybe_future(
  File "C:\Users\Simon\anaconda3\lib\site-packages\tornado\gen.py", line 762, in run
    value = future.result()
  File "C:\Users\Simon\anaconda3\lib\site-packages\notebook\services\kernels\kernelmanager.py", line 176, in start_kernel
    kernel_id = await maybe_future(self.pinned_superclass.start_kernel(self, **kwargs))
  File "C:\Users\Simon\anaconda3\lib\site-packages\jupyter_client\multikernelmanager.py", line 186, in start_kernel
    km.start_kernel(**kwargs)
  File "C:\Users\Simon\anaconda3\lib\site-packages\jupyter_client\manager.py", line 341, in start_kernel
    self.kernel = self._launch_kernel(kernel_cmd, **kw)
  File "C:\Users\Simon\anaconda3\lib\site-packages\jupyter_client\manager.py", line 249, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "C:\Users\Simon\anaconda3\lib\site-packages\jupyter_client\launcher.py", line 132, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "C:\Users\Simon\anaconda3\lib\subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "C:\Users\Simon\anaconda3\lib\subprocess.py", line 1311, in _execute_child
    hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect

Kiyoon Kim · Answer 7 · Tue Feb 14 2023 12:10:49 GMT+0800 (China Standard Time)

For 1. did you start jupyter notebook by yourself, or are you letting Jupynium to start it for you?

If you started by yourself, it will just open localhost:8888 and there's no way it can know where the notebook root directory you set to, so it won't change the directory.

If it started the notebook, it should launch in your file's directory

EDIT: If it asked you for a password it means the notebook server is started from somewhere else. If you close the notebook server and try again, it won't ask you for a password and it will be in your file's directory

Kiyoon Kim · Answer 8 · Tue Feb 14 2023 12:13:07 GMT+0800 (China Standard Time)

For 2. it works on my end. I think your Jupyter Notebook installation is corrupted.

Do you use Conda by any change? In that case I recommend setting the option jupyter_command

Simon Ho · Answer 9 · Tue Feb 14 2023 12:21:40 GMT+0800 (China Standard Time)

I'll take a look at this in the morning, and maybe reinstall anaconda. Every time I run JupyniumStartAndAttachToServer to start things up it always takes me to a page that asks for a password

fyi this is using the base anaconda env, not in a separate env, if that makes a difference

Kiyoon Kim · Answer 10 · Tue Feb 14 2023 12:29:29 GMT+0800 (China Standard Time)

I also use the base environment (so I can switch environments within notebook)

I noticed that the Notebook doesn't close automatically on Windows. I fixed this now btw, but
you should go to firefox/chrome and enter localhost:8888 to see if something's running already.

You can use

ps jupyter-notebook
Stop-Process <id>

to kill the running Notebook process.

Once you do it and update the plugin (still use branch fix/windows) then it will not ask password again.

Kiyoon Kim · Answer 11 · Tue Feb 14 2023 12:50:15 GMT+0800 (China Standard Time)

I'm adding more commits to fix the issue, so when you test it again make sure you update the branch

Kiyoon Kim · Answer 12 · Tue Feb 14 2023 15:27:52 GMT+0800 (China Standard Time)

Please set this and try again. I added support to be more consistent over platforms.

require("jupynium").setup({
  python_host = { "conda", "run", "--no-capture-output", "-n", "base", "python" },
  jupyter_command = { "conda", "run", "--no-capture-output", "-n", "base", "jupyter" },
})

Simon Ho · Answer 13 · Tue Feb 14 2023 17:04:12 GMT+0800 (China Standard Time)

I did a completely fresh install of anaconda, and even wiped my nvim-data directory just for a fresh start. Using the latest on your windows branch and the new config you provided, I still run into the same problem as before. Here is my entire nvim config: https://github.com/sho-87/dotfiles/blob/jupynium/nvim/lua/plugins/modules/jupynium.lua

I'll try to be specific about my steps...

Problem 1 - trying to get basic syncing working

Start Terminal, cd Desktop -> nvim
:JupyniumStartAndAttachToServer ... firefox opens and notebook homepage loads
:JupyniumStartSync ... a new tab and notebook open (Untitled.ipynb), but it is syncing my nvim contents to the notebook (I expect this is the opposite of what is meant to happen). Nothing asked me which way I want to sync
The notebook started with a dead kernel and the same error as I posted above

Problem 2 - trying to get basic syncing working

Start Terminal, cd Desktop -> nvim
:JupyniumStartAndAttachToServer ... firefox opens and notebook homepage loads
In firefox, click on an existing notebook file. Renders fine in firefox, but with dead kernel as above
:JupyniumStartSync 2 to try to sync the new notebook tab to nvim ... I get a warning that my notebook contents will be removed. I expect it's wanting to replace the notebook with my nvim contents again:

Problem 3 - trying to load an existing notebook

Start Terminal, cd Desktop -> nvim
:JupyniumStartAndAttachToServer ... firefox opens and notebook homepage loads
In firefox, I open a notebook file that is on my desktop
:JupyniumLoadFromIpynbTabAndStartSync 2
nvim is frozen and command doesnt seem to get executed. Nothing happens in firefox and nvim remains unresponsive
Notebook page has the same dead kernel and error as above

The reason I tried (3) was because I remember I had syncing working at some point before the anaconda uninstall. I could scroll in nvim and I would see the change in firefox. I could even add cells and clear output; the problem was the dead kernel prevented me from running anything

So I guess my 2 overarching problems are 1) I don't know how to correctly "open" a notebook that already exists, and 2) how to get the kernel to run/restart so code can execute

Jupyter seems fine by itself. I can jupyter notebook and open files with a perfectly running kernel. This problem only comes up when I try do it through jupynium

Kiyoon Kim · Answer 14 · Tue Feb 14 2023 19:15:39 GMT+0800 (China Standard Time)

By the way, this plugin only synchronises from neovim to notebook, not the other way around.
You can load from notebook first, using :JupyniumLoadFromIpynbTabAndStartSync 2, but this will synchronise from neovim to notebook after.
The whole point of this plugin is to edit within neovim, but still see the visualisation on the notebook.
I will fix the problem 3 soon.
About the dead kernel:
Can you clarify if jupyter notebook is fine when you do it after conda activate? The program is merely running a code you configured which is conda run --no-capture-output -n base jupyter notebook ...
Also, when you launch it from the command line which kernel do you get?

Is it Python 3 (ipykernel) or something like Conda ...

Kiyoon Kim · Answer 15 · Tue Feb 14 2023 21:27:31 GMT+0800 (China Standard Time)

I cannot reproduce the problem 2 and 3.

I tried to also uninstall miniconda and install it again. The kernel and the command both works fine for me.. Maybe the dead kernel stuff is causing both problems

Can you try running conda run --no-capture-output -n base jupyter notebook and see if you also see dead kernel if the command is executed that way?

Kiyoon Kim · Answer 16 · Tue Feb 14 2023 21:31:06 GMT+0800 (China Standard Time)

Btw, this is my entire init.lua

require("jupynium").setup({
	python_host = { "conda", "run", "--no-capture-output", "-n", "base", "python" },
	jupyter_command = { "conda", "run", "--no-capture-output", "-n", "base", "jupyter" },
})

And I just cloned this plugin at C:\Users\Kiyoon Kim\AppData\Local\nvim-data\site\pack\dev\start\jupynium.nvim (use fix/windows branch)

Maybe consider testing like this in a fresh environment. I doubt that the kernel error is related to the neovim configuration though. I think it's more related to how you installed jupyter notebook.

For me, after fresh installing miniconda3 I ran

conda activate
conda install python=3.10
pip install notebook
cd "C:\Users\Kiyoon Kim\AppData\Local\nvim-data\site\pack\dev\start\jupynium.nvim"
pip install -e .

Kiyoon Kim · Answer 17 · Tue Feb 14 2023 22:58:47 GMT+0800 (China Standard Time)

You said running jupyter notebook manually doesn't have the kernel problem so another advice would be to open the jupyter notebook server by yourself and try Jupynium. It's less convenient but still usable like that, and maybe it will solve other problems you've had. I will keep trying to fix the kernel problem in the meantime.

Simon Ho · Answer 18 · Tue Feb 14 2023 23:37:14 GMT+0800 (China Standard Time)

conda activate -> jupyter notebook works fine. I only get 1 kernel option, which is Python 3 (ipykernel):

conda run --no-capture-output -n base jupyter notebook also runs ok
using a barebones config leads to the same problem

(all of the above was with the latest commits you pushed)

One big difference here is that youre using miniconda, whereas im using the full anaconda stack: https://www.anaconda.com/products/distribution

Does anaconda install jupyter in some special way beyond simply a pip install?

Kiyoon Kim · Answer 19 · Tue Feb 14 2023 23:44:24 GMT+0800 (China Standard Time)

Thank you. If you run notebook and then use Jupynium, do you still have the problem 3?

Simon Ho · Answer 20 · Tue Feb 14 2023 23:46:09 GMT+0800 (China Standard Time)

what would be the correct commands in Jupynium if im starting the notebook myself first?

Kiyoon Kim · Answer 21 · Tue Feb 14 2023 23:47:14 GMT+0800 (China Standard Time)

Same command. It will just not launch the notebook if you already have localhost:8888 running. It will only load it when it fails to load that page

Kiyoon Kim · Answer 22 · Wed Feb 15 2023 00:07:14 GMT+0800 (China Standard Time)

I've never used anaconda but in my understanding they should be similar. I installed notebook with command pip install notebook but I don't know what will happen if you use conda install notebook.
In any case, the problem is that it does work normally and it doesn't when it's launched with Jupynium. Strange.. Maybe the parameters I passed are not supported with your notebook.

Hypothesis 1: Are you using Notebook 6.5? Maybe different version doesn't support some parameters.

Can you try running with this? This is the full command Jupynium should use to launch the server

conda run --no-capture-output -n base jupyter notebook --port 8888 --no-browser --NotebookApp.token abcdef --NotebookApp.notebook_dir C:\Users\username

Hypothesis 2: Maybe jupynium is just running the wrong command.

Check 1. Did you uninstall the previous conda environment completely?
Check 2. you should try running conda activate and jupynium --jupyter_command conda run " --no-capture-output" " -n" base jupyter. Then try jupynium again. You'll see the error logs there, it may be helpful.

Simon Ho · Answer 23 · Wed Feb 15 2023 00:16:54 GMT+0800 (China Standard Time)

Running jupyter notebook manually and then :JupyniumStartAndAttachToServer works OK. JupyniumStartSync creates a new untitled notebook with a working kernel

:JupyniumLoadFromIpynbTab 2 and :JupyniumLoadFromIpynbTabAndStartSync 2 are both a bit temperamental as they sometimes freezes nvim, but that sounds like a different problem for now

looks like anaconda comes with notebook 6.4. will try an update to 6.5 soon

>>> importlib.metadata.version('notebook')
'6.4.12'

conda run --no-capture-output -n base jupyter notebook --port 8888 --no-browser --NotebookApp.token abcdef --NotebookApp.notebook_dir C:\Users\Simon\Desktop works ok and the kernel is fine

conda activate -> jupynium --jupyter_command conda run " --no-capture-output" " -n" base jupyter gives me an error as its not a program that I can run:

jupynium : The term 'jupynium' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is
correct and try again.

Kiyoon Kim · Answer 24 · Wed Feb 15 2023 00:20:29 GMT+0800 (China Standard Time)

Okay, because of the last part, it sounds to me that you have two different conda installed. You didn't install jupynium in the new conda environment.

Hypothesis 2: Maybe jupynium is just running the wrong command. You shell's conda and neovim's conda can direct you to a different conda command.

Check 3. Run :call system("conda run --no-capture-output -n base jupyter notebook") within your neovim and see if it's still wrong. Check the conda version and install path etc. by calling :echo system("conda -V"), :echo system("conda list")

Kiyoon Kim · Answer 25 · Wed Feb 15 2023 00:22:28 GMT+0800 (China Standard Time)

Or go to cmd.exe instead of powershell and see if your conda environment matches that of powershell.

Kiyoon Kim · Answer 26 · Wed Feb 15 2023 00:25:12 GMT+0800 (China Standard Time)

You can also try setting the neovim's shell to powershell.exe and that may use the correct conda you're using as well.

Kiyoon Kim · Answer 27 · Wed Feb 15 2023 00:25:50 GMT+0800 (China Standard Time)

About the freezing part I'll need to investigate later.

Simon Ho · Answer 28 · Wed Feb 15 2023 00:33:49 GMT+0800 (China Standard Time)

:call system("conda run --no-capture-output -n base jupyter notebook") - kernel works ok
conda 23.1.0
echo system("conda list") # packages in environment at C:\Users\Simon\anaconda3
I dont see jupynium in the conda list, but when I do :echo system("pip freeze") I see jupynium: jupynium @ file:///C:/Users/Simon/AppData/Local/nvim-data/lazy/jupynium.nvim
using cmd.exe, the conda env is at the same location and gives all the same output as 3 and 4

Kiyoon Kim · Answer 29 · Wed Feb 15 2023 00:39:17 GMT+0800 (China Standard Time)

Maybe that's not the case then..
Can you run jupynium like this?

conda activate
python -m jupynium --jupyter_command conda run " --no-capture-output" " -n" base jupyter

Simon Ho · Answer 30 · Wed Feb 15 2023 00:48:41 GMT+0800 (China Standard Time)

ok i think this is a miniconda vs anaconda difference. I just completely uninstalled anaconda and installed a fresh miniconda, and went through these steps:

conda activate
conda install python=3.10
pip install notebook
cd "C:\Users\Simon\AppData\Local\nvim-data\lazy\jupynium.nvim"
pip install -e .

In nvim, JupyniumStartAndAttachToServer and then JupyniumStartSync leads to a working kernel

I'm not sure what the differences are in terms of how miniconda and anaconda install packages (pip vs conda, or something else), but off the bat there are differences in python (3.9 vs 3.10) and package versions (notebook 6.4 vs 6.5)

Kiyoon Kim · Answer 31 · Wed Feb 15 2023 00:57:55 GMT+0800 (China Standard Time)

I see. Thanks for testing that setup and glad that it works! If you're happy I'll merge the PR later today.

In my experience sometimes the conda environment breaks. For example I noticed recently that the conda install doesn't install pytorch with GPU, even though I used the same command as before. Maybe they have some consistency and stability issues.

Also the hanging issue seems to be caused from invoking javascript to grab the content from the notebook. Not sure when it happens maybe I can add a timeout to the command so you wait only up to 10 seconds. If you find this keep happening feel free to open another issue.

Simon Ho · Answer 32 · Wed Feb 15 2023 01:13:25 GMT+0800 (China Standard Time)

unfortunately I cant actually use miniconda permanently, as the stack at work uses the full anaconda setup - so ill have to try to see what the differences between the 2 actually are and figure out a solution

Kiyoon Kim · Answer 33 · Wed Feb 15 2023 01:19:34 GMT+0800 (China Standard Time)

Fair enough. Maybe if you try matching the version it may work. The worst case you'll have to open the Notebook server by yourself but I think that's managable.

Kiyoon Kim · Answer 34 · Wed Feb 15 2023 05:48:37 GMT+0800 (China Standard Time)

This got automatically closed by merging, but feel free to continue the discussion.