Giters

containers / podman

Podman: A tool for managing OCI containers and pods.

Home Page:https://podman.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Running containers in user namespace as root causes overlayfs errors

j0057 opened this issue · comments

commented

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

After having run containers with --uidmap and --gidmap as root, podman image ls throws errors. This situation persists until images are removed.

Steps to reproduce the issue:

  1. Remove all existing images: podman image rm -a
  2. Run a container as root and clean it up again: podman run --rm alpine:3.11 true
  3. Output of podman image ls displays one image, alpine:3.11, and no errors
  4. Run another container as root, but with --uidmap/--gidmap: podman run --rm --uidmap 0:200000:1000 --gidmap 0:200000:1000 alpine:3.11 true
  5. Output of podman image ls displays errors

Describe the results you received:

Output of step 5 podman image ls:

ERRO[0000] error unmounting /var/lib/containers/storage/overlay/08b90647363a5a1bee16a66a176994234707acb754237ed6635729fa08f82d3d/merged: invalid argument 
REPOSITORY                 TAG    IMAGE ID       CREATED       SIZE
docker.io/library/alpine   3.11   a187dde48cd2   3 weeks ago   unable to determine size: size/digest of layer with ID "08b90647363a5a1bee16a66a176994234707acb754237ed6635729fa08f82d3d" could not be calculated: error creating overlay mount to /var/lib/containers/storage/overlay/08b90647363a5a1bee16a66a176994234707acb754237ed6635729fa08f82d3d/merged: stale NFS file handle

Additionally, during step 5 errors are logged in journal:

Apr 14 22:12:24 muon kernel: overlayfs: failed to verify upper (08b90647363a5a1bee16a66a176994234707acb754237ed6635729fa08f82d3d/diff, ino=7226444, err=-116)
Apr 14 22:12:24 muon kernel: overlayfs: failed to verify index dir 'upper' xattr
Apr 14 22:12:24 muon kernel: overlayfs: try deleting index dir or mounting with '-o index=off' to disable inodes index.

Describe the results you expected:

No error messages :-)

Additional information you deem important (e.g. issue happens only occasionally):

My root partition (containing /var) is on a btrfs filesystem on a LUKS volume.

Relevant lines from mount when container like in stap 3 is running:

/dev/mapper/muon_root on /var/lib/containers/storage/overlay type btrfs (rw,relatime,ssd,space_cache,subvolid=5,subvol=/var/lib/containers/storage/overlay)
shm on /var/lib/containers/storage/overlay-containers/2b4bc33d48528151910bd118023e8b0930c95c8790b2aa02be553ab134bc6dd2/userdata/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=64000k)
overlay on /var/lib/containers/storage/overlay/73073bc66e6cc243c91191da6af001aff43a826f7ec22d8d9e4e3cc859395e68/merged type overlay (rw,nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/WMW5T7YZBN2BSCNWHNMFC4CWNH:/var/lib/conta
iners/storage/overlay/l/WMW5T7YZBN2BSCNWHNMFC4CWNH/../diff1:/var/lib/containers/storage/overlay/l/FEPVDLB76XQGTPIGJIGQIUEON3,upperdir=/var/lib/containers/storage/overlay/73073bc66e6cc243c91191da6af001aff43a826f7ec22d8d9e4e3cc859395e68/diff,workdir=/var/lib/containers/storage/overlay/73073bc66e6cc243c91191da6af001aff43a826f7ec22d8d9e4e3cc859395e68/work)

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.14
Git Commit:         028e3317eb1494b9b2acba4a0a295df80fae66cc
Built:              Sat Mar 21 14:30:34 2020
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: 028e3317eb1494b9b2acba4a0a295df80fae66cc
  go version: go1.14
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: 1bddbf7051a973f4a4fecf06faa0c48e82f1e9e1'
  Distribution:
    distribution: arch
    version: unknown
  MemFree: 25248526336
  MemTotal: 33559465984
  OCIRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  SwapFree: 34357637120
  SwapTotal: 34357637120
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: muon
  kernel: 5.6.3-arch1-1
  os: linux
  rootless: false
  uptime: 56m 59.15s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 0
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

# pacman -Q podman
podman 1.8.2-1

Additional environment details (AWS, VirtualBox, physical, etc.):

OS: Arch Linux 
Kernel: x86_64 Linux 5.6.3-arch1-1
Uptime: 1h 1m
Packages: 1072
Shell: bash 5.0.16
Resolution: 3840x2160
WM: i3
GTK Theme: Adwaita [GTK3]
Disk: 541G / 938G (58%)
CPU: Intel Core i7-7700 @ 8x 4.2GHz [43.0°C]
GPU: Intel Corporation HD Graphics 630 (rev 04)
RAM: 2413MiB / 32004MiB
commented

When you are rootless you only have access to the UIDs within your user namespace.

Run another container as root, but with --uidmap/--gidmap: podman run --rm --uidmap 0:200000:1000 --gidmap 0:200000:1000 alpine:3.11 true

200000 is not in the range of UIDs that you own.

This command will work with rootful, or you could attempt a range within the 65k UIDs assigned to your user in /etc/subuid and /etc/subgid.

commented

I'm sorry, I think I may have my terminology wrong. All the steps to reproduce are ran as root, so that makes them rootful containers, even though they use --uidmap/gidmap?

If I run these steps, as root...

  1. podman image rm -a
  2. podman run --rm --uidmap 0:60000:1000 --gidmap 0:60000:1000 -it alpine:3.11 true (so now with a uid/gid less than 65536)
  3. podman image ls

...then the last step, to list the images on my system, throws these errors:

ERRO[0000] error unmounting /var/lib/containers/storage/overlay/379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d/merged: invalid argument 
REPOSITORY                 TAG    IMAGE ID       CREATED       SIZE
docker.io/library/alpine   3.11   a187dde48cd2   3 weeks ago   unable to determine size: size/digest of layer with ID "379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d" could not be calculated: error creating overlay mount to /var/lib/containers/storage/overlay/379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d/merged: stale NFS file handle

As well as some errors in my journal:

Apr 15 13:26:55 muon kernel: overlayfs: failed to verify upper (379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d/diff, ino=7229475, err=-116)
Apr 15 13:26:55 muon kernel: overlayfs: failed to verify index dir 'upper' xattr
Apr 15 13:26:55 muon kernel: overlayfs: try deleting index dir or mounting with '-o index=off' to disable inodes index.
commented

docker.io/library/alpine 3.11 a187dde48cd2 3 weeks ago unable to determine size: size/digest of layer with ID "379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d" could not be calculated: error creating overlay mount to /var/lib/containers/storage/overlay/379557b8dd3e4df8ab2a5bd0afec5cb234777ad438c3c672a0f6f8763d2e5f5d/merged: stale NFS file handle

are you using NFS?

commented

Hi giuseppe, no, this is my filesystem layout:

$ lsblk -f
NAME          FSTYPE      FSVER LABEL     UUID                                 FSAVAIL FSUSE% MOUNTPOINT
nvme0n1                                                                                       
├─nvme0n1p1   crypto_LUKS 1               bcde92a6-ada4-4c99-96a0-847cc43baf8d                
│ └─muon_boot ext4        1.0   muon_boot a443df44-3bcb-476d-9707-64457c32e80a  208.8M    35% /boot
├─nvme0n1p2   vfat        FAT32 muon_efi  B6EA-5671                               509M     0% /boot/efi
├─nvme0n1p3   crypto_LUKS 1               0bf93125-3579-4762-ac6f-cdad7e1d2e7c                
│ └─muon_root btrfs             muon_root 6b80a12e-d7dd-46fb-8e20-24412e8a1fc3    219G    14% /
├─nvme0n1p4   crypto_LUKS 1               8685e186-f563-4d9d-8281-cac5f36465f7                
│ └─muon_home btrfs             muon_home 3f2071e4-69b2-425c-8ed5-3b16074850ff  161.9G    76% /home
└─nvme0n1p5   crypto_LUKS 1               5bd4ae35-b523-4058-8403-883a97cfaa85                
  └─muon_swap swap        1     muon_swap b49bdeec-eebd-45fd-9504-c1128187195e                [SWAP]

It seems like err=-116 from the kernel logs is an error code that is being incorrectly translated to the error about NFS.

commented

When I look at strace -s 1000 -f --failed podman image ls, I see this:

[pid 106761] newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/diff2", 0xc000383488, 0) = -1 ENOENT (No such file or directory)
[pid 106761] newfstatat(AT_FDCWD, "/var/lib/containers/storage/overlay/l/FEPVDLB76XQGTPIGJIGQIUEON3/../diff1", 0xc000383628, 0) = -1 ENOENT (No such file or directory)
[pid 106761] mount("overlay", "/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/merged", "overlay", MS_NODEV, "lowerdir=/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/diff1:/var/lib/containers/storage/overlay/l/FEPVDLB76XQGTPIGJIGQIUEON3,upperdir=/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/diff,workdir=/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/work") = -1 ESTALE (Stale file handle)
[pid 106761] umount2("/var/lib/containers/storage/overlay/6426bc2ddaa4237a844b85c4e4ca35c0db92c77d6e94c4a0fb0f32ad56c1c179/merged", 0) = -1 EINVAL (Invalid argument)

And ESTALE does correspond to 116 in /usr/include/asm-generic/errno.h:

grep -R ESTALE /usr/include/asm-generic/errno.h 
#define	ESTALE		116	/* Stale file handle */

Not sure where the "NFS" in the message comes from.

commented

thanks. Took a while to understand what was going on but I finally managed to reproduce with index=on. I've opened a PR to address it:

containers/storage#598

commented

Thanks for looking into it @giuseppe and @rhatdan!