Girder does not restart in Docker
Billuc opened this issue · comments
Following this discussion on the forum, I wanted to bring you a problem I have encountered.
I am working with Docker containers and one task I've had recently is to create dev Dockers that would automatically update when a change is made to the code (in this case a plugin's code).
In order to do this, I copied my plugin in the container, built the server, mounted the same plugin at the same adsress with a volume and started the server with the --dev option.
The autoreloader is correctly started. However, when I modify the server stops, tries to restart and never does. Do you have an idea of what could cause this ? I have looked at similar issues, but it doesn't look like this issue has been encountered before...
Here is my repro.
Here are the logs I get in my docker :
girder-1 | [26/Feb/2024:10:17:37] ENGINE Restarting because /app/girder/plugin/girder_plugin/test_rest.py changed.
girder-1 | [26/Feb/2024:10:17:37] ENGINE Stopped thread 'Autoreloader'.
girder-1 | [26/Feb/2024:10:17:37] ENGINE Bus STOPPING
girder-1 | [26/Feb/2024:10:17:38] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
girder-1 | [26/Feb/2024:10:17:38] ENGINE Bus STOPPED
girder-1 | [26/Feb/2024:10:17:38] ENGINE Bus EXITING
girder-1 | Stopped asynchronous event manager thread.
girder-1 | [26/Feb/2024:10:17:38] ENGINE Bus EXITED
girder-1 | [26/Feb/2024:10:17:38] ENGINE Waiting for child threads to terminate...
girder-1 | [26/Feb/2024:10:17:38] ENGINE Re-spawning /usr/bin/python3 /usr/local/bin/girder serve --dev
This is fixed in v5, FWIW. #3523
@zachmullen That's great news ! Any idea of when that could come out ?
I can't seem to make it work with v3.2.2, so I am excited for this ! 😃
We don't have an official date but I would expect sometime in the next month or two. Prerelease will likely be within the next couple of weeks.
I just ran a test with a docker I am using. Girder version is 3.2.2.
Girder actually restarts when a change is made !
The problem is that it takes 10 min for Girder to restart, which explains why I thought it didn't restart !
Any idea of why it takes so much time ?
Here are some logs :
girder-dev-1 | Running in mode: development
girder-dev-1 | Connecting to MongoDB: mongodb://mongodb:27017/girder
girder-dev-1 | Loaded plugin "advanced_management"
girder-dev-1 | WARNING: No description docs present for route PUT collection/:/i/d
girder-dev-1 | WARNING: No access level specified for route PUT collection/:/i/d
girder-dev-1 | Loaded plugin "aimodules_ohif"
girder-dev-1 | Loaded plugin "auth_ohif"
girder-dev-1 | Loaded plugin "resources_ohif"
girder-dev-1 | /usr/local/lib/python3.8/site-packages/pymongo/collection.py:1696: UserWarning: use an explicit session with no_cursor_timeout=True otherwise the cursor may still timeout after 30 minutes, for more info see https:
//docs.mongodb.com/v4.4/reference/method/cursor.noCursorTimeout/#session-idle-timeout-overrides-nocursortimeout
girder-dev-1 | return Cursor(self, *args, **kwargs)
girder-dev-1 | [29/Mar/2024:13:11:35] ENGINE Listening for SIGTERM.
girder-dev-1 | [29/Mar/2024:13:11:35] ENGINE Listening for SIGHUP.
girder-dev-1 | [29/Mar/2024:13:11:35] ENGINE Listening for SIGUSR1.
girder-dev-1 | [29/Mar/2024:13:11:35] ENGINE Bus STARTING
girder-dev-1 | [29/Mar/2024:13:11:35] ENGINE Started monitor thread 'Autoreloader'.
girder-dev-1 | Started asynchronous event manager thread.
girder-dev-1 | [29/Mar/2024:13:11:36] ENGINE Serving on http://0.0.0.0:8080
girder-dev-1 | [29/Mar/2024:13:11:36] ENGINE Bus STARTED
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Restarting because /girder/plugins/auth_ohif/girder_auth_ohif/routes/authenticate.py changed.
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Stopped thread 'Autoreloader'.
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Bus STOPPING
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('0.0.0.0', 8080)) shut down
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Bus STOPPED
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Bus EXITING
girder-dev-1 | Stopped asynchronous event manager thread.
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Bus EXITED
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Waiting for child threads to terminate...
girder-dev-1 | [29/Mar/2024:13:35:24] ENGINE Re-spawning /usr/local/bin/python3.8 /usr/local/bin/girder serve --dev
girder-dev-1 | Running in mode: development
girder-dev-1 | Connecting to MongoDB: mongodb://mongodb:27017/girder
girder-dev-1 | Loaded plugin "advanced_management"
girder-dev-1 | WARNING: No description docs present for route PUT collection/:/i/d
girder-dev-1 | WARNING: No access level specified for route PUT collection/:/i/d
girder-dev-1 | Loaded plugin "aimodules_ohif"
girder-dev-1 | Loaded plugin "auth_ohif"
girder-dev-1 | Loaded plugin "resources_ohif"
girder-dev-1 | /usr/local/lib/python3.8/site-packages/pymongo/collection.py:1696: UserWarning: use an explicit session with no_cursor_timeout=True otherwise the cursor may still timeout after 30 minutes, for more info see https:
//docs.mongodb.com/v4.4/reference/method/cursor.noCursorTimeout/#session-idle-timeout-overrides-nocursortimeout
girder-dev-1 | return Cursor(self, *args, **kwargs)
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Listening for SIGTERM.
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Listening for SIGHUP.
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Listening for SIGUSR1.
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Bus STARTING
girder-dev-1 | Started asynchronous event manager thread.
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Started monitor thread 'Autoreloader'.
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Serving on http://0.0.0.0:8080
girder-dev-1 | [29/Mar/2024:13:46:29] ENGINE Bus STARTED
That's definitely odd! I cannot explain it. I've tested on the version 5 integration branch (which is inaptly named v4-integration) and it restarts in about 1-2 seconds in docker-compose.
Alright, I just tried to use the code on branch v4-integration and when I modify the code of my plugin, I still have to wait quite some time... I guess the problem is not directly linked to girder then.
BTW, I used an image that was very similar to yours, so the file system should be pretty much the same !
Here is the Dockerfile :
FROM ubuntu:22.04 AS base
LABEL maintainer="Kitware, Inc. <kitware@kitware.com>"
ENV DEBIAN_FRONTEND=noninteractive \
LANG=en_US.UTF-8 \
LC_ALL=C.UTF-8
RUN apt-get update && apt-get install -qy \
gcc \
libpython3-dev \
git \
libldap2-dev \
libsasl2-dev \
python3-pip \
curl \
&& apt-get clean && rm -rf /var/lib/apt/lists/* \
&& python3 -m pip install --upgrade --no-cache-dir \
pip \
setuptools \
setuptools_scm \
wheel
RUN curl -LJ https://github.com/krallin/tini/releases/download/v0.19.0/tini -o /sbin/tini && \
chmod +x /sbin/tini
RUN curl -sL https://deb.nodesource.com/setup_18.x | bash - && \
apt-get install -qy nodejs
RUN mkdir /girder
# COPY . /girder/
RUN git clone -b v4-integration https://github.com/girder/girder /girder
RUN cd /girder/girder/web && npm i && npm run build
RUN pip install -e /girder
EXPOSE 8080
ENTRYPOINT ["/sbin/tini", "--", "girder", "serve"]
FROM base AS custom
ENV GIRDER_CONFIG /girder/config/girder.cfg
COPY girder.dev.cfg ${GIRDER_CONFIG}
ENV GIRDER_TEST_DB mongodb://mongodb:27017/girder
ENV GIRDER_HOST 0.0.0.0
ENV GIRDER_PORT 8080
COPY plugin /girder/plugin
RUN pip install -e /girder/plugin
ENTRYPOINT ["/sbin/tini", "--", "girder", "serve"]
Hi,
I don't know if this is relevant, but cpu usage by girder in my Docker is 100% during the 10ish minutes between reloads.
I guess something is using a lot of resources during shutdown/startup (maybe ?)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7 root 20 0 7081588 67840 16320 R 100.3 0.2 2:29.93 girder
1 root 20 0 2780 1120 1120 S 0.0 0.0 0:00.02 tini
133 root 20 0 7752 3520 3040 R 0.0 0.0 0:00.01 top