exec in already-running container
jjpr-mit opened this issue · comments
The docs for exec say that unlike Docker's exec, Singularity's exec 'does not require a container to be "running" before using it.' But there is no mention of how to use exec with a container that is already running.
In Docker, I can start a container in one terminal window, and then exec a command (like bash) in the same container from another terminal window. For instance, I can start a Jupyter notebook server in one window, and then use jupyter notebook list from another window, and see my running server.
If I do the same in Singularity, the exec in the second window starts up a new container runtime, separate from the first.
~/dev/tensorflow_scratch$ singularity shell --nv docker://tensorflow/tensorflow:latest-gpu
Singularity tensorflow:latest-gpu:~/dev/tensorflow_scratch> jupyter notebook
[I 14:24:25.581 NotebookApp] Serving notebooks from local directory: /home/someuser/dev/tensorflow_scratch
[I 14:24:25.581 NotebookApp] 0 active kernels
...
In another terminal:
~/dev/tensorflow_scratch$ singularity exec docker://tensorflow/tensorflow:latest-gpu /bin/bash
...
~/dev/tensorflow_scratch$ jupyter notebook list
Currently running servers:
~/dev/tensorflow_scratch$
I don't expect that second command to work; I'm supplying a specification for an image, not the name of a container, so I'm not surprised that it starts a new container from that image. I can't figure out how to name a container. One thing that is confusing me is that the documentation often fails to distinguish between "image" and "container".
You are talking about container orchestration, which is not defined for current release of Singularity. You should look at commands for start and stop in the development branch. The next step would be to add orchestration to work with those.
Okay, good to know.
So, just to confirm, in the current version, is there no way to have two shells into the same container at the same time?
Not easily, but this isn't something I've ever tried to do. You might be able to negotiate some hack of figuring out where the container is mounted, and then issuing commands to content there, bit Ibe never tried it.
Did you try development branch?
I've installed from development commit 5c1d3f0. I'm trying out singularity instance list and singularity instance stop. singularity instance list produces no output but the column headers, even though I have a running container, and I can't use singularity instance stop without knowing the <instance name glob>, which I assume I have to get from singularity instance list? I'm guessing singularity instance start is the same?
I've tried to figure out how Docker does its exec command by reading the source, but I've been unsuccessful. I don't know the project or Go well enough.
Anyway, this issue was just for clarifying the documentation.
heyo! The list command only shows instances you have started and joined, not general containers. If you don't start any instances it will be empty. If you want more of a container manager, then you should check outSingularity Registey (it has a list to show containers)
If you are able to provide a list of commands that you are running, and the expected output, I can try to reproduce the error. I think if you have a running container and you want to exec to it, the uri to reference the container is instance:// although I haven't tried this. If you don't see any instances running (and you did indeed start and join them) then we might have an issue. Anyway, if you could share snippets, that would be hugely useful!
Unfortunately I probably won't be able to work on this for a few weeks.
hey @jjpr-mit ! We are getting ready for 2.4 release docs, any updates on your end?
Sorry, I was offline for a few days. I won't be able to try the dev branch until next week.
I think it would be sufficient to address this issue by changing
Note that this exec is different from the Docker exec, as it does not require a container to be "running" before using it.
to something like
Note that Singularity exec is different from Docker exec: Docker exec runs a command inside an already-running container, while Singularity exec creates a new container instance.
(Unless that's inaccurate.)
A way to think about it is that a Singularity image (when not running as an instance) works exactly as any executable script. Eg: in the same way I can do:
python runme.py
for my container I can do
singularity run runme.img
Running that command does not create a new container "instance" in the way I think you are implying. It executes akin to a script, and the process exits when it's done. We wouldn't want to say "Singularity creates a new container instance" because we are introducing container instances (meaning things with a network namespace that spawn from the container) in 2.4. And starting one of those looks more like:
singularity start runme.img instance-name
and that is entirely different from
singularity run runme.img
Does that make sense?
Heya @jjpr-mit. I think that the thing you are missing is:
singularity instance start some.simg some-name
Try that and then see if you can exec the way you like. 😸
Thank you for your effort. I've installed 2.4 and I'll try it out.
I should clarify that when I used the word "instance" it was not a reference to the new instance feature. It was related to the act of running more than one container based on the same image.