Don't stat unused cgroups; fix for nonexistent pids cgroup?
frobnicaty opened this issue · comments
Description
I'm running current binaries or source-built dockerd, containerd, & runsc on a Synology NAS. (Not using Synology's Docker package.)
Synology doesn't include their kernel configs in the usual places, but according to their partially published source, it has all the Docker required cgroups, and just doesn't support user namespaces. However, there's no /sys/fs/cgroup/perf_event or /sys/fs/cgroup/pids.
perf_event is listed in cgroup.go as a mandatory noop; only rdma is optional. But because you stat the noop perf_event cgroup, which I don't have, it raises an error:
$ docker run --runtime runsc-debug hello-world
docker: Error response from daemon: failed to create shim: OCI runtime create failed: creating container: configuring cgroup: stat /sys/fs/cgroup/perf_event: no such file or directory: unknown.
ERRO[0011] error waiting for container: context canceledI randomly get either that error, or an equivalent one for pids, which is not a noop and I also don't have.
- Why are any of the noops mandatory? Why is a noop
stated at all? If there's not a good reason otherwise, I suggest making all noops optional and totally ignored (or at least ignore failures on things you don't use). - Is there a fix for the lack of a
pidscgroup? The cgroup.go source indicates that it's capable of installing some, but it's not clear to me how this works or how it's invoked, especially if it's not in/proc/cgroups.
$ ~/docker/check-config.sh ~/docker/synokernelconfig.gz
info: reading kernel config from /home/redacted/docker/synokernelconfig.gz ...
Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
Optional Features:
- CONFIG_USER_NS: missing
$ zcat ~/docker/synokernelconfig.gz | grep -i cgroup | sort
# Cgroup
CONFIG_BLK_CGROUP=y
CONFIG_CGROUP_CPUACCT=y
# CONFIG_CGROUP_DEBUG is not set
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
# CONFIG_CGROUP_NET_CLASSID is not set
# CONFIG_CGROUP_NET_PRIO is not set
# CONFIG_CGROUP_PERF is not set
# CONFIG_CGROUP_PIDS is not set
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_WRITEBACK=y
# CONFIG_DEBUG_BLK_CGROUP is not set
# CONFIG_NET_CLS_CGROUP is not set
# CONFIG_NETFILTER_XT_MATCH_CGROUP is not set
CONFIG_SYNO_CGROUP_SKIP_MEMCG=y
$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 8 7 1
cpu 5 244 1
cpuacct 3 244 1
blkio 6 244 1
memory 9 276 1
devices 4 244 1
freezer 7 5 1
$ mount -t cgroup | sort
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/synomonitor type cgroup (rw,nosuid,nodev,noexec,relatime,name=synomonitor)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
$ ls -al /sys/fs/cgroup/
total 0
drwxr-xr-x 11 root root 220 Nov 2 10:34 .
drwxr-xr-x 9 root root 0 Nov 2 10:34 ..
dr-xr-xr-x 45 root root 0 Nov 2 10:34 blkio
dr-xr-xr-x 45 root root 0 Nov 2 10:34 cpu
dr-xr-xr-x 45 root root 0 Nov 2 10:34 cpuacct
dr-xr-xr-x 6 root root 0 Nov 2 10:34 cpuset
dr-xr-xr-x 45 root root 0 Nov 2 10:34 devices
dr-xr-xr-x 5 root root 0 Nov 2 10:34 freezer
dr-xr-xr-x 45 root root 0 Nov 2 10:34 memory
dr-xr-xr-x 42 root root 0 Nov 2 10:34 synomonitor
dr-xr-xr-x 43 root root 0 Nov 2 10:34 systemdSteps to reproduce
No response
runsc version
runsc version release-20211026.0
spec: 1.0.2docker version (if using docker)
$ docker info
Client:
Context: default
Debug Mode: false
Plugins:
compose: Docker Compose (Docker Inc., v2.1.1)
Server:
Containers: 27
Running: 1
Paused: 0
Stopped: 26
Images: 29
Server Version: 20.10.3
Storage Driver: btrfs
Build Version: Btrfs v4.0
Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs db fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc runsc runsc-debug runsc-less runsc-less-ptrace runsc-prof
Default Runtime: runc
Init Binary: docker-init
containerd version: 8686ededfc90076914c5238eb96c883ea093a8ba
runc version: v1.0.2-0-g52b36a2d
init version: de40ad0
Security Options:
apparmor
Kernel Version: 4.4.180+
Operating System: Synology DSM 7.0 (containerized)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 11.66GiB
Name: Redacted
ID: Redacted
Docker Root Dir: /volume1/docker/current/var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true
WARNING: No kernel memory TCP limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No blkio weight support
WARNING: No blkio weight_device support
WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device supportuname
Linux Redacted 4.4.180+ #41890 SMP Thu Jul 15 03:42:45 CST 2021 x86_64 GNU/Linux synology_v1000_1621+
kubectl (if using Kubernetes)
No response
repo state (if built from source)
No response
runsc debug logs (if available)
I1111 03:51:32.273674 22738 main.go:218] ***************************
I1111 03:51:32.273716 22738 main.go:219] Args: [/volume1/docker/current/usr/local/bin/runsc --platform=kvm --overlay --cgroupfs --log=/var/log/runsc.log --network=sandbox --oci-seccomp --debug-log=/tmp/runsc/ --debug --strace --log-packets --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/log.json --log-format json create --bundle /run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195 --pid-file /run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/init.pid 7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195]
I1111 03:51:32.273756 22738 main.go:220] Version release-20211026.0
I1111 03:51:32.273769 22738 main.go:221] GOOS: linux
I1111 03:51:32.273781 22738 main.go:222] GOARCH: amd64
I1111 03:51:32.273793 22738 main.go:223] PID: 22738
I1111 03:51:32.273805 22738 main.go:224] UID: 0, GID: 0
I1111 03:51:32.273817 22738 main.go:225] Configuration:
I1111 03:51:32.273829 22738 main.go:226] RootDir: /var/run/docker/runtime-runc/moby
I1111 03:51:32.273841 22738 main.go:227] Platform: kvm
I1111 03:51:32.273854 22738 main.go:228] FileAccess: exclusive, overlay: true
I1111 03:51:32.273869 22738 main.go:229] Network: sandbox, logging: true
I1111 03:51:32.273884 22738 main.go:230] Strace: true, max size: 1024, syscalls:
I1111 03:51:32.273897 22738 main.go:231] VFS2 enabled: true, LISAFS: false
I1111 03:51:32.273909 22738 main.go:232] Debug: true
I1111 03:51:32.273921 22738 main.go:233] ***************************
W1111 03:51:32.275157 22738 specutils.go:106] AppArmor profile "docker-default" is being ignored
D1111 03:51:32.275437 22738 specutils.go:74] Spec:
{
"ociVersion": "1.0.2-dev",
"process": {
"user": {
"uid": 0,
"gid": 0
},
"args": [
"/hello"
],
"env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"HOSTNAME=7eeaec292fae"
],
"cwd": "/",
"noNewPrivileges": true,
"apparmorProfile": "docker-default",
"oomScoreAdj": 0
},
"root": {
"path": "/volume1/docker/current/var/lib/docker/btrfs/subvolumes/8633acd09a46ade400c1a5d36b0e80696788668284e4fd459d4c4cf426cf1129"
},
"hostname": "7eeaec292fae",
"mounts": [
{
"destination": "/proc",
"type": "proc",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/proc",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev",
"type": "tmpfs",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/tmpfs",
"options": [
"nosuid",
"strictatime",
"mode=755",
"size=65536k"
]
},
{
"destination": "/dev/pts",
"type": "devpts",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/devpts",
"options": [
"nosuid",
"noexec",
"newinstance",
"ptmxmode=0666",
"mode=0620",
"gid=5"
]
},
{
"destination": "/sys",
"type": "sysfs",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/sysfs",
"options": [
"nosuid",
"noexec",
"nodev",
"ro"
]
},
{
"destination": "/sys/fs/cgroup",
"type": "cgroup",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/cgroup",
"options": [
"ro",
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev/mqueue",
"type": "mqueue",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/mqueue",
"options": [
"nosuid",
"noexec",
"nodev"
]
},
{
"destination": "/dev/shm",
"type": "tmpfs",
"source": "/run/containerd/io.containerd.runtime.v2.task/moby/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/shm",
"options": [
"nosuid",
"noexec",
"nodev",
"mode=1777",
"size=67108864"
]
},
{
"destination": "/etc/resolv.conf",
"type": "bind",
"source": "/volume1/docker/current/var/lib/docker/containers/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/resolv.conf",
"options": [
"rbind",
"rprivate"
]
},
{
"destination": "/etc/hostname",
"type": "bind",
"source": "/volume1/docker/current/var/lib/docker/containers/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/hostname",
"options": [
"rbind",
"rprivate"
]
},
{
"destination": "/etc/hosts",
"type": "bind",
"source": "/volume1/docker/current/var/lib/docker/containers/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195/hosts",
"options": [
"rbind",
"rprivate"
]
}
],
"hooks": {},
"linux": {
"resources": {
"memory": {
"disableOOMKiller": false
},
"cpu": {
"shares": 0
},
"blockIO": {
"weight": 0
}
},
"cgroupsPath": "/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195",
"namespaces": [
{
"type": "mount"
},
{
"type": "network"
},
{
"type": "uts"
},
{
"type": "pid"
},
{
"type": "ipc"
}
]
}
}
D1111 03:51:32.275462 22738 container.go:179] Create container, cid: 7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195, rootDir: "/var/run/docker/runtime-runc/moby"
D1111 03:51:32.275568 22738 container.go:237] Creating new sandbox for container, cid: 7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195
D1111 03:51:32.275580 22738 cgroup.go:340] New cgroup for pid: self, &{Name:/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195 Parents:map[] Own:map[]}
D1111 03:51:32.275626 22738 cgroup.go:348] Installing cgroup path "/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.275741 22738 cgroup.go:404] Creating cgroup "memory": "/sys/fs/cgroup/memory/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
I1111 03:51:32.275821 22738 cgroup.go:376] Skipping cgroup "net_prio", err: stat /sys/fs/cgroup/net_prio: no such file or directory
D1111 03:51:32.275838 22738 cgroup.go:404] Creating cgroup "freezer": "/sys/fs/cgroup/freezer/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.275887 22738 cgroup.go:404] Creating cgroup "cpuset": "/sys/fs/cgroup/cpuset/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
I1111 03:51:32.276152 22738 cgroup.go:376] Skipping cgroup "hugetlb", err: stat /sys/fs/cgroup/hugetlb: no such file or directory
D1111 03:51:32.276171 22738 cgroup.go:414] Deleting cgroup "/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.276182 22738 cgroup.go:422] Removing cgroup controller for key="cpuset" path="/sys/fs/cgroup/cpuset/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.276208 22738 cgroup.go:422] Removing cgroup controller for key="memory" path="/sys/fs/cgroup/memory/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.276226 22738 cgroup.go:422] Removing cgroup controller for key="freezer" path="/sys/fs/cgroup/freezer/docker/7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195"
D1111 03:51:32.276443 22738 container.go:705] Destroy container, cid: 7eeaec292fae3eef17672a68f827a45e05df43098a05d6f1043f4a0d3b476195
W1111 03:51:32.276550 22738 error.go:48] FATAL ERROR: creating container: configuring cgroup: stat /sys/fs/cgroup/perf_event: no such file or directory
W1111 03:51:32.276655 22738 main.go:258] Failure to execute command, err: 1