google / crfs

Some container images use whiteouts to indicate "removed entries". But currently, when we use CRFS with overlayfs these whiteouts don't work and no entry is removed.

Assume we have the lower layer:

lower/etc
├── group
├── hostname
├── hosts
├── localtime
├── mtab -> /proc/mounts
├── network
│   ├── if-down.d
│   ├── if-post-down.d
│   ├── if-pre-up.d
│   └── if-up.d
├── passwd
├── resolv.conf
└── shadow

And the upper layer including whiteouts:

upper
└── etc
    ├── network
    │   ├── newfile
    │   └── .wh..wh..opq
    └── .wh.localtime

According to "whiteout" definition in the OCI image specification, the merged directory should be the following(compatible with docker images).

merged/etc
├── group
├── hostname
├── hosts
├── mtab -> /proc/mounts
├── network
│   └── newfile
├── passwd
├── resolv.conf
└── shadow

1 directory, 8 files

But currently CRFS shows these ".wh."-prefixed whiteout files as-is. This behaviour doesn't make overlayfs happy because overlayfs has a different convention to express whiteouts. So it currently results in the following unexpected result:

merged/etc
├── group
├── hostname
├── hosts
├── localtime
├── mtab -> /proc/mounts
├── network
│   ├── if-down.d
│   ├── if-post-down.d
│   ├── if-pre-up.d
│   ├── if-up.d
│   ├── newfile
│   └── .wh..wh..opq
├── passwd
├── resolv.conf
├── shadow
└── .wh.localtime

This bug will be present (kind of) in stargzify if you use the -flatten flag, I think:

crfs/stargz/stargzify/stargzify.go

Line 233 in 1259948

r := mutate.Extract(img)

... since we don't support the ..opq syntax upstream in our flattening code: https://github.com/google/go-containerregistry/blob/50b26ee28691d22901fa3f55c52bdedf6c6a0be1/pkg/v1/mutate/mutate.go#L159

I'm a little confused about the bug & PR. I had assumed since CRFS wasn't doing the merging itself (and is just letting overlayfs do it), all we had to do was faithfully represent each layer's filesystem (whiteout gunk and all) and overlayfs would be happy.

But it's our job to map OCI whiteout spec to overlayfs's whiteout spec?

What normally does that in a non-CRFS world? I guess the code that untars it all to disk?

Anyway, TIL.

@jonjohnsonjr, are you able to review that PR since you seem to know this a bit better than I do?

But it's our job to map OCI whiteout spec to overlayfs's whiteout spec?

It's pretty unfortunate, yeah :( See opencontainers/image-spec#24. I believe the convention comes from AUFS.

What normally does that in a non-CRFS world? I guess the code that untars it all to disk?

I believe so, e.g. here's containerd's approach: https://github.com/containerd/containerd/blob/985bba61ff46c5a24f4a5345f6d1da8c63568bad/archive/tar.go#L120

are you able to review that PR

I'll take a look.

Whiteouts don't work with overlayfs