nipreps / smriprep

Describe the bug

sMRIPrep (0.13.1) fails on autorecon1 and reports that it is unable to find recon-all when run through singularity

Exact command line executed

singularity build code/smriprep_0.13.1.sif docker://nipreps/smriprep:0.13.1
singularity run -e \
  --bind /dcs04/smart/data/K99R00_aphasia:/ds \
  --bind /fastscratch/myscratch/pssadil:/scratch \
  code/smriprep_0.13.1.sif \
  --nprocs 16 \
  --omp-nthreads 8 \
  --mem-gb 24 \
  --output-spaces T1w MNI152NLin2009cAsym MNI152NLin6Asym \
  --fs-license-file /ds/smriprep/code/license \
  -w /scratch/work \
  --participant-label AAO1062 \
  --no-submm-recon \
  /ds/bids/rawdata /ds/smriprep/derivatives participant

Are you positive that the input dataset is BIDS-compliant?

I have used the online BIDS-Validator
I have run a local installation of the BIDS-Validator (please, note the version of the validator here).
I let sMRIPrep check it for me (in other words, I didn't set the --skip-bids-validation argument).
No, I haven't checked myself AND used the --skip-bids-validation argument.

sMRIPrep feedback information
Please attach the full log written to the standard output and the crashfile(s), if generated.

smriprep_0.log
crash-20231128-181647-pssadil-autorecon1-2987f787-6ae9-438e-8502-9740c9643f7b.txt

Visual reports of sMRIPrep
If applicable, add screenshots or attach the corresponding "reportlets" to help explain your problem.

Installation type (please complete the following information):

"Bare-metal"
Singularity
Docker

Additional context
Add any other context about the problem here.

Not sure, but maybe the following is informative?

$ singularity exec code/smriprep_0.13.1.sif recon-all --help
FATAL:   exec /opt/freesurfer/bin/recon-all failed: a shared library is likely missing in the image

$ singularity shell code/smriprep_0.13.1.sif 
Singularity> type recon-all
recon-all is /opt/freesurfer/bin/recon-all
Singularity> exit

$ singularity exec code/smriprep_0.12.2.sif recon-all --help | head

USAGE: recon-all

 Required Arguments:
   -subjid <subjid>
   -<process directive>

 Fully-Automated Directive:
  -all           : performs all stages of cortical reconstruction
  -autorecon-all : same as -all

I don't have an immediate guess for the cause here.

I'm not sure either. Poking around a bit, and I wonder if this could be a lack of tcsh?

❯ docker run --rm -it --entrypoint=/bin/bash nipreps/smriprep:0.13.1
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
(smriprep) root@e82eec9bdff6:/tmp# head -n1 /opt/freesurfer/bin/recon-all 
#! /bin/tcsh -f
(smriprep) root@aaf9a0249adc:/tmp# type tcsh
bash: type: tcsh: not found
(smriprep) root@45cf20997ce7:/tmp# recon-all --help
bash: /opt/freesurfer/bin/recon-all: /bin/tcsh: bad interpreter: No such file or directory
(smriprep) root@aaf9a0249adc:/tmp# exit

❯ docker run --rm -it --entrypoint=/bin/bash nipreps/smriprep:0.12.2
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
(base) root@141bd1f10b4c:/tmp# type tcsh
tcsh is /usr/bin/tcsh

Looks like the Dockerfile changed quite a bit between 0.12.2 and 0.13.1 so maybe this is feasible?

This may be a result of an incompatibility problem between the platform the Docker image was built on (linux/amd64) and your platform (linux/arm64) - perhaps as a result of us moving towards docker buildx.

Could you retry after adding --platform linux/amd64 to your command?

Ah, yes. It looks like we're installing tcsh as part of the AFNI bundle of installs in fMRIPrep, so #387 removed it from the Dockerfile.

Would you be willing to play with re-adding dependencies until we get the necessary ones?

Here's the full set listed in the FreeSurfer 7.3.2 .deb package:

Depends: language-pack-en, binutils, libx11-dev, gettext, xterm, x11-apps, perl, make, csh, tcsh, bash, file, bc, gzip, tar, xorg, xorg-dev, xserver-xorg-video-intel, libncurses5, libbrotli1 (>= 1.0.9), libbsd0 (>= 0.2.0), libc6 (>> 2.35), libc6 (<< 2.36), libcom-err2 (>= 1.43.9), libcrypt1 (>= 1:4.1.0), libdrm2 (>= 2.4.62), libegl1, libexpat1 (>= 2.0.1), libffi8 (>= 3.4), libfontconfig1 (>= 2.12.6), libfreetype6 (>= 2.9.1), libgcc-s1 (>= 4.3), libgl1, libglib2.0-0 (>= 2.22.0), libglu1-mesa | libglu1, libglvnd0, libglx0, libgomp1 (>= 4.9), libgssapi-krb5-2 (>= 1.6.dfsg.2), libice6 (>= 1:1.0.0), libjpeg62 (>= 6b1), libk5crypto3 (>= 1.16), libkeyutils1 (>= 1.5.9), libkrb5-3 (>= 1.19.2), libkrb5support0 (>= 1.18.2), libmd0 (>= 0.0.0), libopengl0, libpcre3, libpng16-16 (>= 1.6.2-1), libquadmath0 (>= 4.6), libsm6, libstdc++6 (>= 5.2), libuuid1 (>= 2.16), libwayland-client0 (>= 1.20.0), libwayland-cursor0 (>= 1.0.2), libx11-6 (>= 2:1.6.0), libx11-xcb1 (>= 2:1.7.5), libxau6 (>= 1:1.0.9), libxcb-icccm4 (>= 0.4.1), libxcb-image0 (>= 0.2.1), libxcb-keysyms1 (>= 0.4.0), libxcb-randr0 (>= 1.3), libxcb-render-util0, libxcb-render0, libxcb-shape0, libxcb-shm0 (>= 1.10), libxcb-sync1, libxcb-util1 (>= 0.4.0), libxcb-xfixes0, libxcb-xinerama0, libxcb-xinput0 (>= 1.14), libxcb-xkb1, libxcb1 (>= 1.12), libxdmcp6, libxext6, libxft2 (>> 2.1.1), libxi6, libxkbcommon-x11-0 (>= 0.5.0), libxkbcommon0 (>= 0.5.0), libxmu6 (>= 2:1.1.3), libxrender1, libxss1, libxt6, zlib1g (>= 1:1.2.11)

I suspect we will just need tcsh and maybe libglu1.

In the short term, you should be able to singularity exec fmriprep.sif smriprep, if you really want to use smriprep instead of fmriprep --anat-only.

This may be a result of an incompatibility problem between the platform the Docker image was built on (linux/amd64) and your platform (linux/arm64) - perhaps as a result of us moving towards docker buildx.

Could you retry after adding --platform linux/amd64 to your command?

@mgxd Just confirming that changing --platform does not help

❯ docker run --rm -it --entrypoint=/bin/bash --platform linux/amd64 nipreps/smriprep:0.13.1
(smriprep) root@bb04563e6108:/tmp# type tcsh
bash: type: tcsh: not found

@effigies , I can experiment with re-adding dependencies and testing manually. I admit that I don't understand the CI setup well enough to be confident, to know why they wouldn't have caught this. It was my impression that the jobs (e.g., ci/circleci: ds005) would have run with an image built from whatever Dockerfile is current. Is that not the case?

recon-all is expensive to run, so we don't run it on CI. Maybe we could remove a final output (maybe stats/wmparc.stats) to ensure -autorecon3 at least gets run. That would catch this error, though that will obviously not catch a library dependency used only by some -autorecon2 stage, which is plausible.

👍🏻 Let me know if I should do that removal (e.g., in this step?).

If you would, please. I would do it immediately before the run:

smriprep/.circleci/config.yml

Lines 457 to 461 in 3b0dcdb

    
                 - run: 
        
                     name: Run anatomical workflow on ds005 
        
                     no_output_timeout: 2h 
        
                     command: | 
        
                       bash /tmp/src/smriprep/.circleci/ds005_run.sh --write-graph

That way we keep the action near its effect.

	- run:
	name: Run anatomical workflow on ds005
	no_output_timeout: 2h
	command: \|
	bash /tmp/src/smriprep/.circleci/ds005_run.sh --write-graph

`recon-all` not detected (v0.13.1) during run with singularity