Read-only file system on postgres:create
AnisSjd opened this issue · comments
Description of problem
I was following the official app deployment tuto at https://dokku.com/docs/deployment/application-deployment/ with my dokku image (from https://hub.docker.com/r/dokku/dokku), but get stuck with:
$ docker compose exec dokku dokku postgres:create railsdatabase
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/railsdatabase/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084
How reproducible
It happens every time on my machine, even when I clean the containers created by dokku.
Steps to Reproduce
echo "postgres: https://github.com/dokku/dokku-postgres.git" > plugin-list;
docker pull dokku/dokku:0.30.2;
docker run -d \
--name dokku-test \
--volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
--volume /var/run/docker.sock:/var/run/docker.sock \
dokku/dokku:0.30.2 &
# sleep 180 # wait Runit to be up
docker exec dokku-test dokku apps:create ruby-getting-started
docker exec dokku-test dokku postgres:create railsdatabase
Actual Results
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/railsdatabase/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084
Expected Results
No error + DB service is created so I can continue the tuto
Environment Information
dokku report output
-----> uname: Linux 0442979ce7de 5.15.0-69-generic #76-Ubuntu SMP Fri Mar 17 17:19:29 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
-----> memory:
total used free shared buff/cache available
Mem: 7759 314 3772 1 3672 7149
Swap: 0 0 0
-----> docker version:
Client: Docker Engine - Community
Version: 23.0.1
API version: 1.41 (downgraded from 1.42)
Go version: go1.19.5
Git commit: a5ee5b1
Built: Thu Feb 9 19:46:56 2023
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 20.10.17
API version: 1.41 (minimum version 1.12)
Go version: go1.18.10
Git commit: a89b842
Built: Tue Mar 7 06:30:14 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.6.6
GitCommit: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc:
Version: 1.1.2
GitCommit:
docker-init:
Version: 0.19.0
GitCommit: de40ad0
-----> docker daemon info:
Client:
Context: default
Debug Mode: true
Plugins:
compose: Docker Compose (Docker Inc.)
Version: v2.16.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 4
Running: 1
Paused: 0
Stopped: 3
Images: 10
Server Version: 20.10.17
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
runc version:
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
cgroupns
Kernel Version: 5.15.0-69-generic
Operating System: Ubuntu Core 18
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.578GiB
Name: vps-9a76c2a9
ID: X7CQ:6YVP:FU5N:QEZS:C7YE:E5H7:J4NQ:63GM:C7LC:DIBZ:PILJ:5TXB
Docker Root Dir: /var/snap/docker/common/var-lib-docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-----> git version: git version 2.25.1
-----> sigil version: 0.9.0build+bc921b7
-----> herokuish version:
Unable to find image 'gliderlabs/herokuish:latest-20' locally
latest-20: Pulling from gliderlabs/herokuish
06d39c85623a: Pulling fs layer
1aebbe1b55dc: Pulling fs layer
cbb9b6b01f72: Pulling fs layer
3d5ce0cd749c: Pulling fs layer
3d5ce0cd749c: Waiting
98cab72ffa42: Pulling fs layer
a786c984824a: Pulling fs layer
98cab72ffa42: Waiting
d66737854f1e: Pulling fs layer
6dba6e891c6b: Pulling fs layer
132bb2747060: Pulling fs layer
70f4c885c73c: Pulling fs layer
d66737854f1e: Waiting
6dba6e891c6b: Waiting
132bb2747060: Waiting
70f4c885c73c: Waiting
1aebbe1b55dc: Download complete
cbb9b6b01f72: Verifying Checksum
cbb9b6b01f72: Download complete
06d39c85623a: Verifying Checksum
06d39c85623a: Download complete
3d5ce0cd749c: Verifying Checksum
3d5ce0cd749c: Download complete
a786c984824a: Verifying Checksum
a786c984824a: Download complete
d66737854f1e: Verifying Checksum
d66737854f1e: Download complete
132bb2747060: Verifying Checksum
132bb2747060: Download complete
6dba6e891c6b: Verifying Checksum
6dba6e891c6b: Download complete
70f4c885c73c: Verifying Checksum
70f4c885c73c: Download complete
06d39c85623a: Pull complete
1aebbe1b55dc: Pull complete
cbb9b6b01f72: Pull complete
3d5ce0cd749c: Pull complete
98cab72ffa42: Verifying Checksum
98cab72ffa42: Download complete
98cab72ffa42: Pull complete
a786c984824a: Pull complete
d66737854f1e: Pull complete
6dba6e891c6b: Pull complete
132bb2747060: Pull complete
70f4c885c73c: Pull complete
Digest: sha256:2f46394d60dbf5839339b7f592596fbaed377201cfe63d37e434689424de6dfb
Status: Downloaded newer image for gliderlabs/herokuish:latest-20
herokuish: v0.5.41
buildpacks:
heroku-buildpack-multi v1.2.0
heroku-buildpack-ruby v252
heroku-buildpack-nodejs v203
heroku-buildpack-clojure v90
heroku-buildpack-python v226
heroku-buildpack-java v72
heroku-buildpack-gradle v39
heroku-buildpack-scala v96
heroku-buildpack-play v26
heroku-buildpack-php v231
heroku-buildpack-go v171
heroku-buildpack-nginx v22
buildpack-null v3
-----> dokku version: dokku version 0.30.2
-----> plugn version: plugn: 0.12.0build+3a27594
-----> dokku plugins:
00_dokku-standard 0.30.2 enabled dokku core standard plugin
20_events 0.30.2 enabled dokku core events logging plugin
app-json 0.30.2 enabled dokku core app-json plugin
apps 0.30.2 enabled dokku core apps plugin
builder 0.30.2 enabled dokku core builder plugin
builder-dockerfile 0.30.2 enabled dokku core builder-dockerfile plugin
builder-herokuish 0.30.2 enabled dokku core builder-herokuish plugin
builder-lambda 0.30.2 enabled dokku core builder-lambda plugin
builder-null 0.30.2 enabled dokku core builder-null plugin
builder-pack 0.30.2 enabled dokku core builder-pack plugin
buildpacks 0.30.2 enabled dokku core buildpacks plugin
caddy-vhosts 0.30.2 enabled dokku core caddy-vhosts plugin
certs 0.30.2 enabled dokku core certificate management plugin
checks 0.30.2 enabled dokku core checks plugin
common 0.30.2 enabled dokku core common plugin
config 0.30.2 enabled dokku core config plugin
cron 0.30.2 enabled dokku core cron plugin
docker-options 0.30.2 enabled dokku core docker-options plugin
domains 0.30.2 enabled dokku core domains plugin
enter 0.30.2 enabled dokku core enter plugin
git 0.30.2 enabled dokku core git plugin
haproxy-vhosts 0.30.2 enabled dokku core haproxy-vhosts plugin
logs 0.30.2 enabled dokku core logs plugin
network 0.30.2 enabled dokku core network plugin
nginx-vhosts 0.30.2 enabled dokku core nginx-vhosts plugin
plugin 0.30.2 enabled dokku core plugin plugin
postgres 1.33.0 enabled dokku postgres service plugin
proxy 0.30.2 enabled dokku core proxy plugin
ps 0.30.2 enabled dokku core ps plugin
registry 0.30.2 enabled dokku core registry plugin
repo 0.30.2 enabled dokku core repo plugin
resource 0.30.2 enabled dokku core resource plugin
run 0.30.2 enabled dokku core run plugin
scheduler 0.30.2 enabled dokku core scheduler plugin
scheduler-docker-local 0.30.2 enabled dokku core scheduler-docker-local plugin
scheduler-null 0.30.2 enabled dokku core scheduler-null plugin
shell 0.30.2 enabled dokku core shell plugin
ssh-keys 0.30.2 enabled dokku core ssh-keys plugin
storage 0.30.2 enabled dokku core storage plugin
trace 0.30.2 enabled dokku core trace plugin
traefik-vhosts 0.30.2 enabled dokku core traefik-vhosts plugin
How (deb/make) and where (AWS, VirtualBox, physical, etc.) was Dokku installed?:
Where: macOS Ventura 13.3 @ Macbook Air M1
How: Docker image at https://hub.docker.com/r/dokku/dokku
Additional information
- Output of failing Dokku commands after running
dokku trace:on: I put only a partial output here, for readability reasons.
If asked I can post the full one.
+ service-create-cmd postgres:create railsdatabase
+ declare 'desc=create a Postgres service'
+ argv=("$@")
+ local cmd=postgres:create argv
+ [[ postgres:create == \p\o\s\t\g\r\e\s\:\c\r\e\a\t\e ]]
+ shift 1
+ CREATE_FLAGS_LIST=("${@:2}")
+ declare SERVICE=railsdatabase CREATE_FLAGS_LIST
+ service_create railsdatabase
+ local SERVICE=railsdatabase
+ is_valid_service_name railsdatabase
+ declare 'desc=validate a service name'
+ declare SERVICE=railsdatabase
+ [[ -z railsdatabase ]]
+ [[ railsdatabase =~ ^[A-Za-z0-9_-]+$ ]]
+ return 0
+ [[ -z railsdatabase ]]
+ [[ ! -d /var/lib/dokku/services/postgres/railsdatabase ]]
+ SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
+ LINKS_FILE=/var/lib/dokku/services/postgres/railsdatabase/LINKS
+ service_parse_args
+ declare 'desc=cli arg parser'
+ local next_index=1
+ local skip=false
+ args=("$@")
+ local args
+ OPTIND=1
+ getopts na:c:C:d:i:I:m:n:N:p:P:q:R:r:s:S:u: opt
+ shift 0
+ service_image_exists railsdatabase
+ declare 'desc=check if the current image exists'
+ declare SERVICE=railsdatabase PLUGIN_IMAGE=postgres PLUGIN_IMAGE_VERSION=15.2
+ local plugin_image=postgres
+ local plugin_image_version=15.2
+ [[ -z postgres ]]
+ [[ -z 15.2 ]]
+ local IMAGE=postgres:15.2
++ /usr/local/bin/docker image ls -q postgres:15.2
+ [[ 80c558ffdc31 == '' ]]
+ return 0
+ plugn trigger service-action pre-create postgres railsdatabase
+ mkdir -p /var/lib/dokku/services/postgres/railsdatabase
+ mkdir -p /var/lib/dokku/services/postgres/railsdatabase/data
+ touch /var/lib/dokku/services/postgres/railsdatabase/LINKS
++ openssl rand -hex 16
+ PASSWORD=bff386bea5a5dd69f51c1d77794f677a
+ [[ -n '' ]]
+ echo bff386bea5a5dd69f51c1d77794f677a
+ chmod 640 /var/lib/dokku/services/postgres/railsdatabase/PASSWORD
+ service_commit_config railsdatabase
+ declare SERVICE=railsdatabase
+ local SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
+ local CONFIG_VARIABLE=POSTGRES_CONFIG_OPTIONS
+ local ENV_VARIABLE=POSTGRES_CUSTOM_ENV
+ custom_env=
+ [[ -n '' ]]
+ [[ -n '' ]]
+ echo ''
+ config_options=
+ [[ -n '' ]]
+ [[ -n '' ]]
+ echo ''
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n postgres ]]
+ echo postgres
+ [[ -n 15.2 ]]
+ echo 15.2
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ write_database_name railsdatabase
+ declare 'desc=write a sanitized database name'
+ declare SERVICE=railsdatabase
+ local SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
+ echo railsdatabase
+ tr .- _
+ plugn trigger service-action post-create postgres railsdatabase
+ service_create_container railsdatabase
+ local SERVICE=railsdatabase
+ local SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
+ local SERVICE_HOST_ROOT=/var/lib/dokku/services/postgres/railsdatabase
++ get_service_name railsdatabase
++ declare 'desc=retrieve a docker service label'
++ declare SERVICE=railsdatabase
++ echo dokku.postgres.railsdatabase
+ local SERVICE_NAME=dokku.postgres.railsdatabase
++ service_password railsdatabase
++ declare 'desc=fetch the password for a service'
++ declare SERVICE=railsdatabase
++ local SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
++ local PASSWORD_FILE=/var/lib/dokku/services/postgres/railsdatabase/PASSWORD
++ [[ -f /var/lib/dokku/services/postgres/railsdatabase/PASSWORD ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/PASSWORD
+ local PASSWORD=bff386bea5a5dd69f51c1d77794f677a
++ get_database_name railsdatabase
++ declare 'desc=retrieve a sanitized database name'
++ declare SERVICE=railsdatabase
++ local SERVICE_ROOT=/var/lib/dokku/services/postgres/railsdatabase
++ [[ ! -f /var/lib/dokku/services/postgres/railsdatabase/DATABASE_NAME ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/DATABASE_NAME
+ local DATABASE_NAME=railsdatabase
+ local PREVIOUS_ID
+ [[ -f /var/lib/dokku/services/postgres/railsdatabase/CONFIG_OPTIONS ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/CONFIG_OPTIONS
+ export CONFIG_OPTIONS=
+ CONFIG_OPTIONS=
++ service_dns_hostname railsdatabase
++ declare 'desc=retrieve the alias of a service'
++ declare SERVICE=railsdatabase
+++ get_service_name railsdatabase
+++ declare 'desc=retrieve a docker service label'
+++ declare SERVICE=railsdatabase
+++ echo dokku.postgres.railsdatabase
++ local SERVICE_NAME=dokku.postgres.railsdatabase
++ tr ._ -
++ echo dokku.postgres.railsdatabase
+ local network_alias=dokku-postgres-railsdatabase
+ rm -f /var/lib/dokku/services/postgres/railsdatabase/ID
+ declare -a DOCKER_ARGS
+ DOCKER_ARGS=()
+ DOCKER_ARGS+=("--cidfile=$SERVICE_ROOT/ID")
+ DOCKER_ARGS+=("--env-file=$SERVICE_ROOT/ENV")
+ DOCKER_ARGS+=("--env=POSTGRES_PASSWORD=$PASSWORD")
+ DOCKER_ARGS+=("--hostname=$SERVICE_NAME")
+ DOCKER_ARGS+=("--label=dokku.service=$PLUGIN_COMMAND_PREFIX")
+ DOCKER_ARGS+=("--label=dokku=service")
+ DOCKER_ARGS+=("--name=$SERVICE_NAME")
+ DOCKER_ARGS+=("--restart=always")
+ DOCKER_ARGS+=("--volume=$SERVICE_HOST_ROOT/data:/var/lib/postgresql/data")
+ declare -a LINK_CONTAINER_DOCKER_ARGS
+ LINK_CONTAINER_DOCKER_ARGS=()
+ LINK_CONTAINER_DOCKER_ARGS+=("--rm")
+ LINK_CONTAINER_DOCKER_ARGS+=("--link")
+ LINK_CONTAINER_DOCKER_ARGS+=("$SERVICE_NAME:$network_alias")
+ [[ -f /var/lib/dokku/services/postgres/railsdatabase/SERVICE_MEMORY ]]
+ [[ -n '' ]]
+ [[ -f /var/lib/dokku/services/postgres/railsdatabase/SHM_SIZE ]]
+ [[ -n '' ]]
+ [[ -f /var/lib/dokku/services/postgres/railsdatabase/IMAGE ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/IMAGE
+ PLUGIN_IMAGE=postgres
+ [[ -f /var/lib/dokku/services/postgres/railsdatabase/IMAGE_VERSION ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/IMAGE_VERSION
+ PLUGIN_IMAGE_VERSION=15.2
++ fn-plugin-property-get postgres railsdatabase initial-network
++ declare 'desc=returns the value for a given property'
++ declare PLUGIN=postgres APP=railsdatabase KEY=initial-network DEFAULT=
++ fn-plugin-property-get-default postgres railsdatabase initial-network ''
++ declare 'desc=returns the value for a given property with a specified default value'
++ declare PLUGIN=postgres APP=railsdatabase KEY=initial-network DEFAULT=
++ /var/lib/dokku/core-plugins/available/common/prop get-with-default postgres railsdatabase initial-network ''
+ local network=
+ [[ -n '' ]]
+ suppress_output /usr/local/bin/docker container create --cidfile=/var/lib/dokku/services/postgres/railsdatabase/ID --env-file=/var/lib/dokku/services/postgres/railsdatabase/ENV --env=POSTGRES_PASSWORD=bff386bea5a5dd69f51c1d77794f677a --hostname=dokku.postgres.railsdatabase --label=dokku.service=postgres --label=dokku=service --name=dokku.postgres.railsdatabase --restart=always --volume=/var/lib/dokku/services/postgres/railsdatabase/data:/var/lib/postgresql/data postgres:15.2
+ declare 'desc=suppress all output from a given command unless there is an error'
+ local TMP_COMMAND_OUTPUT
++ mktemp /tmp/dokku-8069-suppress_output.XXXXXX
+ TMP_COMMAND_OUTPUT=/tmp/dokku-8069-suppress_output.RKBdFu
+ trap 'rm -rf '\''/tmp/dokku-8069-suppress_output.RKBdFu'\'' >/dev/null' RETURN
+ /usr/local/bin/docker container create --cidfile=/var/lib/dokku/services/postgres/railsdatabase/ID --env-file=/var/lib/dokku/services/postgres/railsdatabase/ENV --env=POSTGRES_PASSWORD=bff386bea5a5dd69f51c1d77794f677a --hostname=dokku.postgres.railsdatabase --label=dokku.service=postgres --label=dokku=service --name=dokku.postgres.railsdatabase --restart=always --volume=/var/lib/dokku/services/postgres/railsdatabase/data:/var/lib/postgresql/data postgres:15.2
+ return 0
++ rm -rf /tmp/dokku-8069-suppress_output.RKBdFu
++ fn-plugin-property-get postgres railsdatabase post-create-network
++ declare 'desc=returns the value for a given property'
++ declare PLUGIN=postgres APP=railsdatabase KEY=post-create-network DEFAULT=
++ fn-plugin-property-get-default postgres railsdatabase post-create-network ''
++ declare 'desc=returns the value for a given property with a specified default value'
++ declare PLUGIN=postgres APP=railsdatabase KEY=post-create-network DEFAULT=
++ /var/lib/dokku/core-plugins/available/common/prop get-with-default postgres railsdatabase post-create-network ''
+ [[ -n '' ]]
++ cat /var/lib/dokku/services/postgres/railsdatabase/ID
+ suppress_output /usr/local/bin/docker container start eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084
+ declare 'desc=suppress all output from a given command unless there is an error'
+ local TMP_COMMAND_OUTPUT
++ mktemp /tmp/dokku-8069-suppress_output.XXXXXX
+ TMP_COMMAND_OUTPUT=/tmp/dokku-8069-suppress_output.U5EGnF
+ trap 'rm -rf '\''/tmp/dokku-8069-suppress_output.U5EGnF'\'' >/dev/null' RETURN
+ /usr/local/bin/docker container start eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084
+ local exit_code=1
+ cat /tmp/dokku-8069-suppress_output.U5EGnF
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/railsdatabase/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: eedca8efbc4bde92824a379e4455a2a70fad8caec4c0e11490cc11ea9638d084
+ return 1
++ rm -rf /tmp/dokku-8069-suppress_output.U5EGnF
What host os are you running Dokku on?
Host: macOS Ventura 13.3
Docker image: Linux 0442979ce7de 5.15.0-69-generic (from https://hub.docker.com/r/dokku/dokku)
The docker installation notes show environment variables and mounts that need to be set. The host directories should be writeable - if not, you'll get this error. I think thats expected on OSX, so you'll need to use a host path other than /var/lib/dokku for data.
You are right, I used the instructions at https://hub.docker.com/r/dokku/dokku, instead of the (more complete?) ones at https://dokku.com/docs/getting-started/install/docker/ .
I fixed my previous command, but the issue remains.
Note that I intentionally haven't mounted my host /var/lib/dokku to avoid bind mount issues as the one you mentionned.
Here is the updated command:
docker run --rm \
--env DOKKU_HOSTNAME=dokku.me \
--env DOKKU_HOST_ROOT=/var/lib/dokku/home/dokku \
--env DOKKU_LIB_HOST_ROOT=/var/lib/dokku/var/lib/dokku \
--name dokku-test \
--publish 3022:22 \
--publish 8080:80 \
--publish 8443:443 \
--volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
--volume /var/run/docker.sock:/var/run/docker.sock \
dokku/dokku:0.30.2 &
The datastore plugins store data on disk and use the mounted docker socket, so all the directories are mounted from the host.
The host root env vars are used to specify where to mount directories from. Since you set them relative to /var/lib/dokku, that's where data gets mounted from.
OK, the following params let me create the database:
docker run --rm \
--env DOKKU_HOSTNAME=dokku.me \
--env DOKKU_HOST_ROOT=${PWD}/home/dokku \
--env DOKKU_LIB_HOST_ROOT=${PWD}/var/lib/dokku \
--name dokku-test \
--publish 3022:22 \
--publish 8080:80 \
--publish 8443:443 \
--volume ${PWD}/var/lib/dokku:/mnt/dokku \
--volume ${PWD}/plugin-list:/mnt/dokku/plugin-list \
--volume /var/run/docker.sock:/var/run/docker.sock \
dokku/dokku:0.30.2 &
I understand now the importance of DOKKU_LIB_HOST_ROOT; however I'm still unsure of its exact use and limitations, since I can't find enough documentation about it, both at https://hub.docker.com/r/dokku/dokku and https://dokku.com/docs/getting-started/install/docker/.
Thanks for help.
@josegonzalez I am running into the same issue. However I am running on Ubuntu server, from the bootstrap.sh.
My docker report is here
sudo dokku postgres:create sample_db
Error response from daemon: error while creating mount source path '/var/lib/dokku/services/postgres/sample_db/data': mkdir /var/lib/dokku: read-only file system
Error: failed to start containers: 835486d8356fff70d64cf8ca1da254db161f9567212d5766643869f6060fdfd3
I managed to resolve the issue by uninstalling the snap docker (snap remove docker) and reinstalling docker following https://docs.docker.com/engine/install/ubuntu/ then rebooting.