Watch with pull and recreate
lonix1 opened this issue · comments
As originally discussed here. I was asked by @glours to open this separate issue to keep track of it.
@lonix wrote
Hi @ndeloof I was directed here from the blog post.
The blog states that the draft spec consists of actions: "sync" and "rebuild+recreate".
There is another action which just as important, and FAR simpler than those: "recreate only".
In a CI/CD environment:
- the new image will be built and uploaded to some registry
- the running app must then be restarted with the new version
- the app probably has the current version specified in
.envordocker-compose.yml, so that file will be updated by the CD tooling - docker compose should simply detect that file was changed, and recreate the container... that's all!
I saw you going back and forth on this with some doubters in other threads. But this new watch experiment is a brilliant idea. Maybe consider this use case as the foot in the door - it's simple to explain and justify, and sorely needed. Doing CD with docker is hard, but a watch-restart feature would make it trivial.
Please consider giving us a "restart only" action - I would use it in production immediately and toss out half a dozen scripts and tools and kludges!
Thanks!
@nemchik wrote
There is another action which just as important, and FAR simpler than those: "restart only".
Just to clarify, you're looking for a watch with a pull and recreate, not just a restart, correct?
@lonix1 wrote
Yes, you are right, it's a pull + recreate - the equivalent of docker compose down && docker compose up.
To eliminate all that stuff for a native and dead-simple docker compose watch feature would be a game changer. Complex CI/CD toolchains would be much shorter and easier to deploy and manage.
Another use case here is if the running process uses a file that was changed but cannot pick up on those changes until it's been restarted. I've been running into this issue lately and a restart only feature would be amazing!
If I understood this issue correctly, there is even a much more common case and need for this.
A compose.yaml is often used in development. Lets pretend we have a service postgres. One developer updates the Postgres version we support and thus make an updated in compose.yaml to use the new Docker image tag for the new version. When other developers rebase/merge (sync their local branch with upstream) we want docker compose to automatically update that developers running service. Basically like having a watch on compose.yaml itself (as well as env files or even env vars).
This would possibly be good in a branch-switching context. If I just rebased one local branch and move between others, the other branches should possibly still used the old version of Postgres as other application code might depend on it. So the service would then be updated again if switching branch.
Often use cases:
- sources mutated → rebuild (takes time)
- full rebuild (bypassing cache)
- rebuild using cache
- sources mutated → need to sync and restart
- if user is not root then "sync+restart" fails
- if user is "root" then it works
- sources mutated → need to recreate (different image) but sync is external
- pull+recreate (missing)
- sources mutated → need to restart (same image) but sync is external
- resources (either "src" or "dist") are mounted as a volume so no need to sync
Possible workarounds:
- use nodemon to watch container files (mounted volume) and restart only the process
- no workaround for externally pushed image :(
@lonix1 what you describe here is a mechanism to watch for updated images, not local file. Please note the develop section is used for this feature, this is not intended to keep your production server up to date as a CI/CD magic pipeline :)
Anyway, maybe it would make sense we offer an option to recreate container (not just restart) and also pull+recreate to cover this use-case.
Basically like having a watch on compose.yaml itself
this is a chicken/egg challenge we have to address here, as updating compose file would require to reload and refresh the watch actions. But that would be nice
Another use case here is if the running process uses a file that was changed but cannot pick up on those changes until it's been restarted. I've been running into this issue lately and a restart only feature would be amazing!
this is supported by sync+restart action
@maciej-wakula-opuscapita "if user is not root then sync+restart fails": can you please elaborate on this limitation ?
@maciej-wakula-opuscapita "if user is not root then sync+restart fails": can you please elaborate on this limitation ?
I have a bit better understanding now - not sure if it is about the "root" permissions.
When a file change is detected then a method is called to tar the files. There is also barely documented boolean env variable COMPOSE_EXPERIMENTAL_WATCH_TAR used implemented here.
To tar you need to specify:
services:
app:
develop:
watch:
- action: 'sync+restart'
path: './src'
# and we are done... right? No "target" needed
Since there is no target - golang executes tar command with incorrect arguments - and fails.
See #460
Second bottom
I found users reporting that this error is related to the user. Like if you have a Dockerfile
FROM ubuntu:latest
WORKDIR /app
COPY --chown root:root ./src /app
RUN chmod 000 -R /app
USER nobody
I found reports of untar command failure when you cannot write to target location. I'll update this comment later today after testing the following:
services:
app:
develop:
watch:
- action: 'sync+restart'
path: './src'
# and we are done... right? No "target" needed
volumes:
- ./src/file.txt:/app/src/file.txt:ro
Finally I think the code is not much readable when it comes to "rebuild", "sync+restart" and "whatever else". I think that all 3 (or more) scenarios should be listed in a switch statement which then calls already implemented pieces.
maciej-wakula-opuscapita here but from private account.
Using Pop!OS (ubuntu based) 22.04
with docker 24.0.5 and docker-compose-plugin 2.20.2+ds1-0ubuntu122.04.1~
@ndeloof I have created multiple test scenarios. Left only key scenarios here.
How I have updated docker and what should be improved in the documentation
There are too many: docker, docker.io, k3d, k3s, podman, etc. Then `docker-compose` (legacy python code) and `docker compose` plugin (new in golang). It is easy to get lost in this.I would expect that docker describes difference between:
- the docker vs docker.io
- docker-compose vs docker-compose-plugin (docker compose)
- docker-compose.yml vs compose.yml
- compose.yml vs compose.override.yml
What I did (using pop!os which is an ubuntu):
I have uninstalled old version: apt remove docker docker.io docker-compose-v2 docker-compose
following docker instructions for ubuntu I downloaded docker-desktop-4.27.1-amd64.deb
and executed sudo apt-get install ./docker-desktop-4.27.1-amd64.deb
sudo apt install qemu-system
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && chmod a+r /etc/apt/keyrings/docker.asc && echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
docker --version
# Docker version 25.0.2, build 29cf629
docker compose version
# Docker Compose version v2.24.3-desktop.1
Then prepared test environment and executed first test
/root/s418/ Dockerfile
FROM ubuntu:22.04
WORKDIR /app
USER nobody
/root/s418/ compose.yml
services:
app:
build: .
develop:
watch:
- action: 'sync+restart'
path: './src'
Test
# First terminal
docker compose down --remove-orphans
docker compose build --no-cache
# [+] Building 0.7s (6/6) FINISHED
docker compose watch
# [+] Building 0.5s (6/6) FINISHED
# [+] Running 1/2
# ⠼ Network s418_default Created # 0.4s
# ✔ Container s418-app-1 Started # Watch configuration for service "app":
# - Action sync+restart for path "/root/s418/src"# Second terminal
touch src/subdir/file.txt# First terminal
#WARN[0025] Error handling changed files for service app: 1 error occurred:
# * copying files to 473b62cb6781880b8336b85a12c3fc955088f9de51e96e864278829d448cb770: Error response from daemon: cannot overwrite directory "/" with non-directory "/"Attempt 2 - with read-only volume underneath
/root/s418/ Dockerfile
FROM ubuntu:22.04
RUN mkdir -p /app/src/subdir
WORKDIR /app
USER nobody
/root/s418/ compose.yml
services:
app:
build: .
develop:
watch:
- action: 'sync+restart'
path: './src'
volumes:
- ./src/subdir:/app/subdir:ro
Test
# First terminal
docker compose down --remove-orphans
docker compose build --no-cache
# [+] Building 1.0s (7/7) FINISHED
docker compose watch
# [+] Building 0.8s (7/7) FINISHED
# [+] Running 1/2
# ⠼ Network s418_default Created # 0.4s
# ✔ Container s418-app-1 Started # Watch configuration for service "app":
# - Action sync+restart for path "/root/s418/src"# Second terminal
touch src/subdir/file.txt# First terminal
#Syncing "app" after changes were detected:
# - /root/s418/src/subdir/file.txt
#WARN[0003] Error handling changed files for service app: 1 error occurred:
# * copying files to 6389ccb88e94526e37f9d9a53dd804524f3404a1186e8bf94193f7965a44abd8: Error response from #daemon: unlinkat /app/src/subdir/file.txt: read-only file system