Pulling latest version of MultiQC image on GitPod fails

[Simon Murray]

Description of bug

When I am inside a GitPod container which has Docker preinstalled and I attempt to pull the MultiQC Docker Image: quay.io/biocontainers/multiqc:1.19--pyhdfd78af_0 or quay.io/biocontainers/multiqc:1.18--pyhdfd78af_0 the image does not get pulled locally but instead produces this error: failed to register layer: "lsetxattr /etc: operation not supported".

However the image quay.io/biocontainers/multiqc:1.17--pyhdfd78af_1 does not produce this error.

File that triggers the error

This is the GitPod yaml I am using:
gitpod.yml.zip

MultiQC Error log

The full command that fails is:
docker pull quay.io/biocontainers/multiqc:1.19--pyhdfd78af_0

The full log is:

1.19--pyhdfd78af_0: Pulling from biocontainers/multiqc
93e509ffbdaf: Pull complete 
820c67f471a9: Extracting [==================================================>]  9.794MB/9.794MB
bd9ddc54bea9: Download complete 
1d3e40e109f2: Download complete 
failed to register layer: lsetxattr /etc: operation not supported

Before submitting

• I have read the troubleshooting documentation.
• I am using the latest release of MultiQC.
• I have included a full MultiQC log, not truncated.
• I have attached an input file (.zip if necessary) that triggers the error.

[Phil Ewels]

Thanks for reporting. This issue came up in nf-core too, it seems to be due to a recent change in Docker itself. Copying the relevant discussion:

[@mashehu] Hello people using docker. The current release of docker engine (25.0.0) unfortunately breaks some of our module setup, resulting in the error: failed to register layer: lsetxattr /etc: operation not supported. https://github.com/nf-core/tools/actions/runs/7611989497/job/20728591406?pr=2677#step:13:200 @mirpedrol dug around and we think it might be related to moby/moby#47153. The best solution for now is to downgrade docker...

[@pontus] It seems to be a problem with extended attributes in some images (e.g. the multiqc), rootless mode and overlay2. At least in my test machine I could get around it (in rootless) by switching to the fuse-overlayfs storage driver instead. I also took a quick stab a gitpod and changing to fuse-overlayfs lets me pull at least, but I see some error when trying to run there (but it's not unlikely to be caused by my dirty restart of the daemon).

Anyhow, I don't think this will affect many users, and for the runner that should be fairly easy to work around (don't use rootless or use fuse-overlayfs). Gitpod would be my biggest worry.

But these attributes causing issues (user.overlay.impure and user.overlay.origin) shouldn't leak in to the images anyway, not sure if this is for all biocontainers images from extended-base or some other subset.

[@mirpedrol] https://github.com/nf-core/actions-runners/pull/8/files fix for the self-hosted runners

I'm not entirely clear on whether there's anything that we can do with the MultiQC biocontainers image. I think it's to do with the biocontainers build backend and the GitPod stack setup, plus the docker engine itself.

Does anyone above have any recommendations for MultiQC Biocontainers + GitPod?

[Pontus Freyhult]

I suspect this might just be a glitch, but don't have insight in the biocontainers build setup (or have been able to find it easily). Most images (those I've checked) do not appear to have this issue.

So, if anyone knows how the docker build system works and if a rebuild can be triggered, that could be worth a shot. Otherwise, if some bugs are squashed so it makes sense to do a release, that would also exercise that.

[Alexander Peltzer]

I also had a bit of a more deeper look into this. Gitpod by default uses an XFS filesystem with attr2 set (the error relates to extended attributes being written for a specific file (/etc) and this seems to fail for a user.:

As you can also see these errors pop up with other projects, I am not 100% sure this is only a biocontainers issue per se but more a combination a.) changes in how docker does this sort of things and b.) gitpod having a filesystem that is not dealing well with this. The code used for creating mulled containers can be found here: https://github.com/galaxyproject/galaxy/tree/dev/lib/galaxy/tool_util/deps/mulled

We can retrigger a rebuild, simply bumping the build number +1 for multiqc on bioconda, no worries - just did this.

[Alexander Peltzer]

Wait a few hours, then we'll see if the new container is fine.

[Phil Ewels]

Given that this seems to have been every build since v1.17, I doubt a rebuild will fix it 😕 (but happy to be proven wrong!).

NB: Should be releasing v1.21 on Monday too. Was planning for today but didn't quite make it.

[Pontus Freyhult]

Created bioconda/bioconda-utils#958 now, I don't know where that image comes from, but it does seem like it's used somewhere, which would cause this issue.

[Simon Murray]

Interestingly the latest version of mutliqc produces a similar but different error when pulling in GitPod:

gitpod /workspace/synteny (main) $ docker pull quay.io/biocontainers/multiqc:1.20--pyhdfd78af_2
1.20--pyhdfd78af_2: Pulling from biocontainers/multiqc
93e509ffbdaf: Pull complete 
820c67f471a9: Extracting [==================================================>]  9.794MB/9.794MB
bd9ddc54bea9: Download complete 
5babece2d55e: Download complete 
failed to register layer: lsetxattr user.overlay.impure /etc: operation not supported

Thanks for all your support!

[Alexander Peltzer]

Ah no, this is the same error (or at least the same reason with lsetxattr).

[Pontus Freyhult]

Yes, I think this is moby/moby#47178 being included in docker engine 25.0.1, so same error, slightly more informative error message.