Is r2u compatible with Singularity?
ginolhac opened this issue · comments
Dear Dirk,
on a HPC machine, we need to use Singularity and not directly docker.
So, first converting the docker image is:
singularity pull docker://eddelbuettel/r2u:jammy
Then, the singularity image (SIF) is created but the admin part was ripped off. I can start a container with R, but the the great CRANapt seems to have suffered from the operation:
$ singularity exec r2u_jammy.sif R
R version 4.2.0 (2022-04-22) -- "Vigorous Calisthenics"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
> install.packages("Seurat")
Error: cannot connect to the system package manager
Am I doing it wrong or is it a limitation? Classic containers of singularity image with R and install.packages() calls are fine.
Many thanks in advance,
I have strictly no idea as I do not use Singularity. The basic idea of r2u is to support install.packages() and friends via apt by relying on bspm whereever the components work. If you find yourself in a case where they do not, you still have apt directly available and get all thise benefits.
I did find, however, that I sometimes (eg initially on AWS, but now apparently also on a Ubuntu 22.04 host) need to add the option --security-opt seccomp=unconfined to the docker invocation. Maybe that is all you need here to 'elevate' or 'clear' your access rights to system process inside the container.
hum thank you. It sounds like a dead end. The apt install works great in docker (just tried). But failed in singularity, as we are not allowed to run anything as admin. I will investigate a bit more but I am not optimistic. Nothing wrong with you repo actually, so I guess we can close.
Singularity is a nice idea, and saying "No thank you" to system level changes can be defensible.
But it does close the door on services offering at the system level such as r2u. Such is life--you win some, you loose some.
As a bioinformatician, Singularity has saved many days. No need to struggle with weird install in my HOME, missing libraires, outdated modules available etc.... I can plan ahead and create with docker a system that works. Using r2u would be the cherry on an already great cake.
Actually I should be able to use your system upfront on the laptop and then create the singularity image. It would have been awesome to have a generic image and fast install but maybe that should work.
Will give a try and report tomorrow, fast install locally, save the image, convert to SIF and use only on HPC.
To end the conversation, this final idea seems great. For example to run Seurat on a HPC via Singularity:
- Create the Docker image locally with
r2u, takes < 20 seconds. Dockerfile:
FROM eddelbuettel/r2u:jammy
LABEL org.opencontainers.image.authors="Aurelien Ginolhac <aurelien.ginolhac@uni.lu>"
RUN apt-get update && \
DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
r-cran-seurat
- Save the docker image:
docker save --output seurat.tar seurat - Scp the tar image to the HPC
- On HPC, build the Singularty image:
singularity build seurat.sif docker-archive://seurat.tar - Test:
singularity exec seurat.sif Rscript -e "library(Seurat)"
The whole process takes < 10 min, quite enjoyable.
Works well this workaround and get fast. Of note, the seurat.tar is 1.1 GB while the seurat.sif is 417 MB so you see how much is removed in Singularity.
"These days" with these containers that have bspm support and the littler scripts I would just do this in 1.
RUN apt update -qq && \
DEBIAN_FRONTEND=noninteractive install.r Seurat
(and you can of course list multiple packages so, say, Seurat brms collapse from three non-overlapping sets also works.)
Rest is fine and the way to go: you make use of root power where you can and then seal things for deployment on Singularity.
(Edited for apt update -qq. Not sure you need the DEBIAN_FRONTEND either as we are not starting from bare-bones, r2u has timezones etc where the nag can come in. Anyway -- this is nice. We should do a short blog post showing with Signac or Seurat or one of those large packages.)
(Another possible edit: bspm also updates apt for you so we can really reduce it to RUN install.r Seurat. Nice and short ;-) )
Thanks! yes indeed, it is better and I will update my way of doing things that are a bit "copy/pasting from the last Dockerfile found"
Just today again Singularity rescued me using gatk (Java framework) where 4 yo script didn't work anymore and recent install neither. But now the broadinstitute provides a docker image. Then I only needed to convert to a Singularity image. It is truly amazing. And actually r2u is just fitting in nicely, only need to remember to polish the docker image with relevant install upfront.
Let me know if you need any input for a blog post.
Thanks again for the great work
Just for the sake of being complete. Singularity is part of Syslabs and was a fork of an intial Singularity that was not a commercial product. Now the latter is joining the Linux Fondation and is rebranded Apptainer. See details here https://apptainer.org/news/community-announcement-20211130