local file mounted as folder in container on flatcar docker swarm node
jdall opened this issue · comments
Description
[ 1 paragraph concisely describing the bug ]
A deploy of SigNoz to a docker swarm node running flatcar 3815.2.3 fails, because apparently some local configuration files are being mounted as folders instead of files in a container.
Impact
[ 1 sentence detailing the impact this bug is creating for you ]
For now the impact is minor, as we are just in the initial phase of testing SigNoz.
Environment and steps to reproduce
-
Set-up: [ describe the environment Flatcar/Lokomotive/Nebraska etc was running in when encountering the bug; Platform etc. ]
The Flatcar instance is running on a bare-metal x86_64 server.
The instance has been initialized as a docker swarm master node. -
Task: [ describe the task performing when encountering the bug ]
I was trying to deploy SigNoz to the swarm node, using docker compose and the docker-compose file provided by the SigNoz GitHub repository. -
Action(s): [ sequence of actions that triggered the bug, see example below ]
a. git clone -b main https://github.com/SigNoz/signoz.git
b. cd signoz/deploy/docker-swarm
c. docker compose -f clickhouse-setup/docker-compose.yaml up -d -
Error: [describe the error that was triggered]
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "****/docker-swarm/signoz/clickhouse-setup/clickhouse-config.xml" to rootfs at "/etc/clickhouse-server/config.xml": mount ****/docker-swarm/signoz/clickhouse-setup/clickhouse-config.xml:/etc/clickhouse-server/config.xml (via /proc/self/fd/7), flags: 0x5000, data: context="system_u:object_r:container_file_t:s0:c392,c505": not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
Expected behavior
[ describe what you expected to happen at 4. above but instead got an error ]
All containers would start running.
Additional information
Please add any information here that does not fit the above format.
The same deployment has been tested towards a docker swarm node running on AlmaLinux 9, where the files are correctly mounted as files in the container, and all containers starts running.
The section in the docker-compose.yaml file, that results in the error, looks as follows:
clickhouse:
<<: *clickhouse-defaults
hostname: clickhouse
# ports:
# - "9000:9000"
# - "8123:8123"
# - "9181:9181"
volumes:
- ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
- ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
- ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
#- ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
- ./data/clickhouse/:/var/lib/clickhouse/
In my debugging, I tried modifying the above section slightly, to have a peek into the container filesystem:
clickhouse:
<<: *clickhouse-defaults
hostname: clickhouse
# ports:
# - "9000:9000"
# - "8123:8123"
# - "9181:9181"
command: "ls -la /etc/clickhouse-server"
volumes:
- ./clickhouse-config.xml:/etc/clickhouse-server/config1.xml
#- ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
#- ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
## - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
#- ./data/clickhouse/:/var/lib/clickhouse/
Which resulted in the following output from the container:
clickhouse-1 | total 124
clickhouse-1 | drwxrwxrwx 1 1000 1000 4096 Jun 22 14:08 .
clickhouse-1 | drwxr-xr-x 1 1000 1000 4096 Jun 22 14:08 ..
clickhouse-1 | drwxrwxrwx 1 root root 4096 Feb 2 11:46 config.d
clickhouse-1 | -rw-rw-rw- 1 1000 1000 78357 Feb 1 13:43 config.xml
clickhouse-1 | drwxr-xr-x 2 root root 4096 Jun 21 20:03 config1.xml
clickhouse-1 | drwxrwxrwx 2 root root 4096 Feb 2 11:46 users.d
clickhouse-1 | -rw-rw-rw- 1 1000 1000 5672 Feb 1 13:43 users.xml
As can be seen, the config1.xml file has been mounted as a directory!!
Docker Engine Info (with docker host running on AlmaLinux 9):
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.45
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:30:53 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 26.1.4
API version: 1.45 (minimum version 1.24)
Go version: go1.21.11
Git commit: de5c9cf
Built: Wed Jun 5 11:29:11 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.33
GitCommit: d2d58213f83a351ca8f528a95fbd145f5654e957
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker Engine Info (with docker host running on Flatcar):
Client: Docker Engine - Community
Version: 26.1.4
API version: 1.43 (downgraded from 1.45)
Go version: go1.21.11
Git commit: 5650f9b
Built: Wed Jun 5 11:30:53 2024
OS/Arch: linux/amd64
Context: default
Server:
Engine:
Version: 24.0.9
API version: 1.43 (minimum version 1.12)
Go version: go1.20.12
Git commit: fca702de7f71362c8d103073c7e4a1d0a467fadd
Built: Tue May 21 11:03:41 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: v1.7.13
GitCommit: 99b8088b873ba42b788f29ccd0dc26ebb6952f1e
runc:
Version: 1.1.12
GitCommit: 51d5e94601ceffbbd85688df1c928ecccbfa4685
docker-init:
Version: 0.19.0
GitCommit: de40ad007797e0dcd8b7126f27bb87401d224240
FYI, the issue appears to also be present when deploying to a flatcar instance which has not been initialized for docker swarm usage.
The steps:
a. git clone -b main https://github.com/SigNoz/signoz.git
b. cd signoz/deploy/docker
c. docker compose -f clickhouse-setup/docker-compose.yaml up
Results in the following error being shown:
Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "***/signoz/deploy/docker/clickhouse-setup/clickhouse-config.xml" to rootfs at "/etc/clickhouse-server/config.xml": mount ***/signoz/deploy/docker/clickhouse-setup/clickhouse-config.xml:/etc/clickhouse-server/config.xml (via /proc/self/fd/7), flags: 0x5000, data: context="system_u:object_r:container_file_t:s0:c336,c904": not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
Hi @jdall, thanks for the details and for the reproduction steps. I can't reproduce yet the issue, can you share those information:
- how did you install docker compose?
- Is the Git repository cloned with a specific user (different from
core
) - between your tests, did you purge everything? (
docker compose -f clickhouse-setup/docker-compose.yaml down -v
) to be sure there is no leftovers from previous tests?
EDIT:
I've done this:
$ git clone -b main https://github.com/SigNoz/signoz.git
$ cd signoz/deploy/docker
$ docker compose -f clickhouse-setup/docker-compose.yaml up clickhouse -d
$ docker compose -f clickhouse-setup/docker-compose.yaml exec clickhouse head /etc/clickhouse-server/config.xml
<?xml version="1.0"?>
<!--
NOTE: User and query level settings are set up in "users.xml" file.
If you have accidentally specified user-level settings here, server won't start.
You can either move the settings to the right place inside "users.xml" file
or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> here.
-->
<clickhouse>
<logger>
<!-- Possible levels [1]:
Thanks, @tormath1
Actually, your question "Is the Git repository cloned with a specific user (different from core)" makes me realize that my problem is probably caused by the fact that I have checked out the code on my workstation, while I'm trying to deploy to a flatcar instance running on a remote host.
So basically, I guess I'm trying to do something that docker compose is not meant to do in the first place :/
Please feel free to close this bug report.
@jdall ah yes, that's possible indeed. What you could do as a workaround would be to upload the files (clickhouse-config.xml
and friends) on Flatcar via initial provisioning (Ignition).
A Systemd unit could take of running the docker compose
command.
Similar to this: #894 (comment)
Do not hesitate to join our Matrix or Slack channel if you want to discuss and explore this method with Flatcar users: https://github.com/flatcar/Flatcar?tab=readme-ov-file#chat