[TRACKER ONLY]: (qm-minimal.mpp.yml) 0.6.2: nested container: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory
dougsland opened this issue · comments
Just found this one: nested container: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory
See-Also: #405
git clone https://gitlab.com/CentOS/automotive/sample-images.git
pushd osbuild-manifests/
make cs9-qemu-qm-minimal-regular.x86_64.qcow2
./runvm --nographics ./cs9-qemu-qm-minimal-regular.x86_64.qcow2
user: root pass: password
[root@localhost ~]# podman exec -it qm bash
bash-5.1# podman run -it --name fedora fedora /bin/bash
Error: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file
or directory
Another test:
bash-5.1# podman ps
Error: overlay: can't stat imageStore dir /usr/share/containers/storage: stat /usr/share/containers/storage: no such file or directory
@Yarboa for some reason, I thought this one was fixed via https://github.com/containers/qm/blob/main/setup#L91-L116
Digging it.
talked with @rhatdan and his debug approach helped to identify the issue and he mentioned that RHEL9.5 should be fixed by default.
Debugging:
bash-5.1# podman --root /usr/share/containers/storage pull alpine
Error: creating runtime static files directory "/usr/share/containers/storage/libpod": mkdir /usr/share/containers/storage: read-only file system
The final solution for this error is (in the host, not in QM):
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images
touch /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images/images.lock
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/libpod
mkdir /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers
touch /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers/layers.lock
Now podman is working as expected
bash-5.1# podman pull alpine
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Writing manifest to image destination
05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd
bash-5.1# podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
bash-5.1# podman images
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/alpine latest 05455a08881e 3 months ago 7.67 MB
bash-5.1# podman info
host:
arch: amd64
buildahVersion: 1.33.5
cgroupControllers:
- cpuset
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: Unknown
path: /usr/bin/conmon
version: 'conmon version 2.1.10, commit: d807bb8c1de3dc05fb66c77d2979a7f6903804bf'
cpuUtilization:
idlePercent: 99.19
systemPercent: 0.54
userPercent: 0.27
cpus: 4
databaseBackend: sqlite
distribution:
distribution: centos
version: "9"
eventLogger: journald
freeLocks: 2048
hostname: localhost
idMappings:
gidmap: null
uidmap: null
kernel: 5.14.0-438.391.el9iv.x86_64
linkmode: dynamic
logDriver: journald
memFree: 1648013312
memTotal: 2060533760
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: Unknown
package: Unknown
path: /usr/libexec/podman/netavark
version: netavark 1.10.3
ociRuntime:
name: crun
package: Unknown
path: /usr/bin/crun
version: |-
crun version 1.14.4
commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
pasta:
executable: ""
package: ""
version: ""
remoteSocket:
exists: false
path: /run/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: Unknown
version: |-
slirp4netns version 1.2.3
commit: c22fde291bb35b354e6ca44d13be181c76a0a432
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.2
swapFree: 0
swapTotal: 0
uptime: 0h 7m 15.00s
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
search:
- registry.access.redhat.com
- registry.redhat.io
- docker.io
store:
configFile: /etc/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions:
overlay.imagestore: /usr/share/containers/storage
overlay.mountopt: nodev,metacopy=on
graphRoot: /var/lib/containers/storage
graphRootAllocated: 7938207744
graphRootUsed: 1022078976
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "true"
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/containers/storage
transientStore: true
volumePath: /var/lib/containers/storage/volumes
version:
APIVersion: 4.9.4-dev
Built: 1710930166
BuiltTime: Wed Mar 20 10:22:46 2024
GitCommit: ""
GoVersion: go1.21.7 (Red Hat 1.21.7-1.el9)
Os: linux
OsArch: linux/amd64
Version: 4.9.4-dev
bash-5.1#
Now I need to figure out our code and see why what we have is not working and prepare a patch.
@dougsland it is all described in the Spike work
setup script is not running while building autosd image (regular/ostree)
in qm setup script, it update the qm:podman cofn files with the relevant data.
this setup is not running inside osbuild
So there are two option to bring it into work
- as setup script update config files
- as suggested above.
Whatever chosen, should work both qm deployments
Anyway option 2, requires minor fix in qm.mpp
@pbrilla-rh This issue might not be a blocker for the release as this behavior only happens in the images/qm-minimal.mpp.yml. If QM minimal is NOT a requirement for the release, we keep this bug opened and move forward with the release.
I have tested the image cs9-qemu-qmcontainer-regular.x86_64.qcow2 (images/qmcontainer.mpp.yml) and it WORKED just fine with podman + nested containers in QM. However, I see another issue with the OSTREE image version: cs9-qemu-qmcontainer-ostree.x86_64.qcow2 (not really related to QM code but still):
# make cs9-qemu-qmcontainer-ostree.x86_64.qcow2
osbuild-mpp -I . -D image_type="\"ostree\"" -D arch=\"x86_64\" -D distro_name="\"cs9\"" -D target="\"qemu\"" images/qmcontainer.mpp.yml _build/cs9-qemu-qmcontainer-ostree.x86_64.json
osbuild --checkpoint build --store _build/osbuild_store --output-directory _build/image_output --cache-max-size=1GB --export qcow2 _build/cs9-qemu-qmcontainer-ostree.x86_64.json
_build/cs9-qemu-qmcontainer-ostree.x86_64.json has errors:
.pipelines[6].stages[5]:
could not find schema information for 'org.osbuild-auto.ostree.pre-gen'
make: *** [Makefile:171: cs9-qemu-qmcontainer-ostree.x86_64.qcow2] Error 2
Is this image requirement for the release? I guess not, as the others releases always went ahead without it... but worth double check right?
I was asking myself why qmcontainer.mpp.yml worked and qm-minimal.mpp.yml not?
What's the difference to make it work?
+ - type: org.osbuild.mkdir
+ options:
+ paths:
+ - path: /usr/share/containers/storage/overlay-images
+ parents: true
+ - path: /usr/share/containers/storage/overlay-layers
+ parents: true
+ - type: org.osbuild.copy
+ inputs:
+ inlinefile:
+ type: org.osbuild.files
+ origin: org.osbuild.source
+ mpp-embed:
+ id: empty
+ path: ../files/empty
+ options:
+ paths:
+ - from:
+ mpp-format-string: input://inlinefile/{embedded['empty']}
+ to: tree:///usr/share/containers/storage/overlay-images/images.lock
+ - from:
+ mpp-format-string: input://inlinefile/{embedded['empty']}
+ to: tree:///usr/share/containers/storage/overlay-layers/layers.lock
Even applying this directly in the qm-mininal yaml file still not working and might explain why this change was only added in qmcontainer.mpp.yml and not in the minimal version in the time. When adding In the qm-minimal during the build it complains about readonly file system during the lock files creation (even few lines above creating the dirs). At least in my test environment.
I also would like to suggest a little change to the solution, move the code to image.ipp.yml (to deliver the same behavior in ALL qm images in a single place) instead of dup the code to qm-minimal until the podman 5 lands in EL9. However, it's also not working for the same reason I mentioned above (the readonly in the lock files). My question for now, should we move this one to the next release pending tasks?
Finally, thanks @Yarboa for always pinging me in the slack offering help.
diff --git a/osbuild-manifests/include/image.ipp.yml b/osbuild-manifests/include/image.ipp.yml
index 4f7d9aa..32872bb 100644
--- a/osbuild-manifests/include/image.ipp.yml
+++ b/osbuild-manifests/include/image.ipp.yml
@@ -125,6 +125,30 @@ mpp-vars:
parents: true
- path: /var/qm/
parents: true
+ # START: workaround until podman 5 is included into the distro
+ # https://github.com/containers/qm/issues/406
+ - path: /usr/lib/qm/usr/share/containers/storage
+ parents: true
+ - path: /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-images
+ parents: true
+ - path: /usr/lib/qm/rootfs/usr/share/containers/storage/overlay-layers
+ parents: true
+ # END: workaround until podman 5 is included into the distro
+ # https://github.com/containers/qm/issues/406
+
+ # START: workaround until podman 5 is included into the distro
+ # https://github.com/containers/qm/issues/406
+ type: org.osbuild.truncate
+ options:
+ filename: /usr/lib/qm/usr/share/containers/storage/overlay-images/images.lock
+ size: '0'
+ type: org.osbuild.truncate
+ options:
+ filename: /usr/lib/qm/usr/share/containers/storage/overlay-layers/layers.lock
+ size: '0'
+ # END: workaround until podman 5 is included into the distro
+ # https://github.com/containers/qm/issues/406
@dougsland
we should update build doc, just install
sudo dnf install -y osbuild-auto
This issue should be managed in the sig, not here
Just confirmed, https://gitlab.com/CentOS/automotive/sample-images/-/merge_requests/506 worked for me (qm-minimal). Probably there are others ways to fix it (like installing osbuild-auto + the patch I mentioned above but let's move forward with the simplest as possible). +1 @Yarboa Thanks! Waiting the review and merge to close this one.
This issue is just a tracker at this moment.
Nothing related to QM, @Yarboa is taking care of it in the sample-image project.