missing `redis.conf` causing redis container crash
maciejopalinski opened this issue · comments
Description of problem
redis.conf is not created in container config directory
container crashes because it has no redis.conf
redis.conf is downloaded to /var/lib/dokku/services/redis/test/config/ in dokku container, but not getting copied to /mnt/dokku/services/redis/test/config/ (that is mounted to host) causing the sibling container to not find redis.conf and crash
Steps to Reproduce
- Run
dokkuin Docker container.
# docker-compose.yml
dokku:
container_name: dokku
restart: unless-stopped
image: dokku/dokku:latest
ports:
- 3022:22
- 8080:80
- 8443:443
volumes:
- /var/lib/dokku:/mnt/dokku
- /var/run/docker.sock:/var/run/docker.sock
env_file:
- .dokku.env# .dokku.env
DOKKU_HOSTNAME = "example.com"
DOKKU_HOST_ROOT = "/var/lib/dokku/home/dokku"
docker exec -it dokku bashsudo dokku plugin:install https://github.com/dokku/dokku-redis.git redisdokku redis:create test(this will fail or not, it's random)
- back to host machine
docker ps- dokku.redis.test will have status restartingdocker logs dokku.redis.test
1:C 21 Oct 2022 21:53:26.473 # Fatal error, can't open config file '/usr/local/etc/redis/redis.conf': No such file or directory
[REDACTED: repeated many times]
- if you check what volumes
dokku.redis.testhas mounted usingdocker inspect dokku.redis.testyou can see, that:
"Mounts": [
{
"Type": "bind",
"Source": "/var/lib/dokku/services/redis/test/data",
"Destination": "/data",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/var/lib/dokku/services/redis/test/config",
"Destination": "/usr/local/etc/redis",
"Mode": "",
"RW": true,
"Propagation": "rprivate"
}
],
- host machine: now, if you check
ls -la /var/lib/dokku/services/redis/test/configit will be empty, so the error inside redis container is totally expected - back to dokku container:
docker exec -it dokku bashls -la /var/lib/dokku/services/redis/test/config-redis.confexistsroot@36cc59e302af:/tmp# ls -la /var/lib/dokku/services/redis/test/config/ total 116 drwxrwxr-x 2 dokku dokku 4096 Oct 21 22:19 . drwxrwxr-x 4 dokku dokku 4096 Oct 21 22:19 .. -rw-rw-r-- 1 dokku dokku 106599 Oct 21 22:19 redis.confls -la /mnt/dokku/services/redis/test/config- HA! it doesn't exist on the mounted volume to the host machineroot@36cc59e302af:/tmp# ls -la /mnt/dokku/services/redis/test/config/ total 8 drwxr-xr-x 2 root root 4096 Oct 21 22:19 . drwxr-xr-x 4 root root 4096 Oct 21 22:19 .. root@36cc59e302af:/tmp#
Conclusion
I assume that the issue is service_commit_config function call or plugn trigger service-action post-create call.
Environment Information
dokku report output
-----> uname: Linux 78df7607037e 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 3827 1088 163 85 2576 2367
Swap: 0 0 0
-----> docker version:
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.16.2
Git commit: 20.10.12-0ubuntu2~20.04.1
Built: Wed Apr 6 02:14:38 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.20
API version: 1.41 (minimum version 1.12)
Go version: go1.18.7
Git commit: 03df974
Built: Tue Oct 18 18:18:08 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.8
GitCommit: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc:
Version: 1.1.4
GitCommit: v1.1.4-0-g5fd4c4d
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Server:
Containers: 16
Running: 14
Paused: 0
Stopped: 2
Images: 20
Server Version: 20.10.20
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-52-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 3.738GiB
Name: [REDACTED: privacy]
ID: OCHO:ESHG:LXWJ:JB2Q:JYPA:SF4B:NEDB:L5PO:WLAU:WIDW:SPLT:LYZA
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> git version: git version 2.25.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
herokuish: 0.5.37
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v244
heroku-buildpack-nodejs v198
heroku-buildpack-clojure v90
heroku-buildpack-python v214
heroku-buildpack-java v72
heroku-buildpack-gradle v38
heroku-buildpack-scala v94
heroku-buildpack-play v26
heroku-buildpack-php v223
heroku-buildpack-go v166
heroku-buildpack-nginx v16
buildpack-null v3
-----> dokku version: dokku version 0.28.3
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
00_dokku-standard 0.28.3 enabled dokku core standard plugin
20_events 0.28.3 enabled dokku core events logging plugin
app-json 0.28.3 enabled dokku core app-json plugin
apps 0.28.3 enabled dokku core apps plugin
builder 0.28.3 enabled dokku core builder plugin
builder-dockerfile 0.28.3 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.28.3 enabled dokku core builder-herokuish plugin
builder-lambda 0.28.3 enabled dokku core builder-lambda plugin
builder-null 0.28.3 enabled dokku core builder-null plugin
builder-pack 0.28.3 enabled dokku core builder-pack plugin
buildpacks 0.28.3 enabled dokku core buildpacks plugin
caddy-vhosts 0.28.3 enabled dokku core caddy-vhosts plugin
certs 0.28.3 enabled dokku core certificate management plugin
checks 0.28.3 enabled dokku core checks plugin
common 0.28.3 enabled dokku core common plugin
config 0.28.3 enabled dokku core config plugin
cron 0.28.3 enabled dokku core cron plugin
docker-options 0.28.3 enabled dokku core docker-options plugin
domains 0.28.3 enabled dokku core domains plugin
enter 0.28.3 enabled dokku core enter plugin
git 0.28.3 enabled dokku core git plugin
logs 0.28.3 enabled dokku core logs plugin
network 0.28.3 enabled dokku core network plugin
nginx-vhosts 0.28.3 enabled dokku core nginx-vhosts plugin
plugin 0.28.3 enabled dokku core plugin plugin
postgres 1.25.0 enabled dokku postgres service plugin
proxy 0.28.3 enabled dokku core proxy plugin
ps 0.28.3 enabled dokku core ps plugin
redis 1.25.0 enabled dokku redis service plugin
registry 0.28.3 enabled dokku core registry plugin
repo 0.28.3 enabled dokku core repo plugin
resource 0.28.3 enabled dokku core resource plugin
run 0.28.3 enabled dokku core run plugin
scheduler 0.28.3 enabled dokku core scheduler plugin
scheduler-docker-local 0.28.3 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.28.3 enabled dokku core scheduler-null plugin
shell 0.28.3 enabled dokku core shell plugin
ssh-keys 0.28.3 enabled dokku core ssh-keys plugin
storage 0.28.3 enabled dokku core storage plugin
trace 0.28.3 enabled dokku core trace plugin
traefik-vhosts 0.28.3 enabled dokku core traefik-vhosts plugin
How (deb/make/rpm) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:
Dokku was installed in a docker container on a Hetzner VPS.
Additional information
- Output of failing Dokku commands after running
dokku trace:on
++ docker images -q redis:7.0.5
+ [[ f8528f17261c == '' ]]
+ return 0
+ plugn trigger service-action pre-create redis test
+ mkdir -p /var/lib/dokku/services/redis/test
+ mkdir -p /var/lib/dokku/services/redis/test/data
+ mkdir -p /var/lib/dokku/services/redis/test/config
+ touch /var/lib/dokku/services/redis/test/LINKS
+ [[ -z '' ]]
+ curl -sSL https://raw.githubusercontent.com/antirez/redis/7.0/redis.conf
++ openssl rand -hex 32
+ PASSWORD=3a32eed448f12cbdde484ca4681bee4e08e5d5311ca77948d8adc450dbcd6504
+ [[ -n '' ]]
+ echo 3a32eed448f12cbdde484ca4681bee4e08e5d5311ca77948d8adc450dbcd6504
+ chmod 640 /var/lib/dokku/services/redis/test/PASSWORD
[REDACTED: some password]
+ rm /var/lib/dokku/services/redis/test/config/redis.conf.bak
[REDACTED: not relevant]
+ PLUGIN_IMAGE_VERSION=7.0.5
++ docker run --name dokku.redis.test -v /var/lib/dokku/services/redis/test/data:/data -v /var/lib/dokku/services/redis/test/config:/usr/local/etc/redis --env-file=/var/lib/dokku/services/redis/test/ENV -d --restart always --label dokku=service --label dokku.service=redis redis:7.0.5 redis-server /usr/local/etc/redis/redis.conf --bind 0.0.0.0
+ ID=b3aeb9d0c7f95319f917fd06c4004011af233a120d30c6ad9855453e5fafdcbd
+ echo b3aeb9d0c7f95319f917fd06c4004011af233a120d30c6ad9855453e5fafdcbd
as you can see in the trace logs, there is a command curl -sSL https://raw.githubusercontent.com/antirez/redis/7.0/redis.conf, so the config is downloaded
@josegonzalez can you verify that the issue is also occurring for you?
I believe setting REDIS_HOST_ROOT=/mnt/dokku/services/redis in the dokku container env vars should fix this.
I believe setting
REDIS_HOST_ROOT=/mnt/dokku/services/redisin thedokkucontainer env vars should fix this.
@josegonzalez This does not solve the issue.
getting the same issue here, the REDIS_HOST_ROOT env var on the dokku container didn't help either.
I think this is related to #5468, but I'll confirm this week.
I think this is related to #5468, but I'll confirm this week.
This should work now with the addition of the DOKKU_LIB_HOST_ROOT environment variable to the launch config of the dokku container. Documentation has been updated in dokku/dokku#5744 and should be rolling out to the docs site shortly.