Different values to `ldd` in Gentoo Prefix
ocaisa opened this issue · comments
I was trying to use libtree
for some testing under Gentoo Prefix. I know we may need to patch CUDA to look in the prefix but I was surprised that ldd
and lddtree
don't return the same information:
[ocaisa@gpunode1 release]$ ldd ./deviceQuery
linux-vdso.so.1 (0x00007ffd78ed6000)
librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000153ce5eac000)
libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000153ce5e8c000)
libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000153ce5e86000)
libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x0000153ce5c71000)
libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x0000153ce5b30000)
libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x0000153ce5b13000)
libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000153ce5958000)
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x0000153ce5ec3000)
[ocaisa@gpunode1 release]$ lddtree ./deviceQuery
deviceQuery => ./deviceQuery (interpreter => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2)
librt.so.1 => /lib64/librt.so.1
libpthread.so.0 => /lib64/libpthread.so.0
libdl.so.2 => /lib64/libdl.so.2
libstdc++.so.6 => /lib64/libstdc++.so.6
libm.so.6 => /lib64/libm.so.6
libgcc_s.so.1 => /lib64/libgcc_s.so.1
libc.so.6 => /lib64/libc.so.6
ld-linux-x86-64.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
[ocaisa@gpunode1 release]$ which ldd
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/bin/ldd
Which is correct here? Is it that libtree
is using the system linker rather than the one specified in the executable?
lddtree
!= libtree
, seems like you're opening an issue in the wrong repo?
Argh, idiot, sorry, I do have libtree
installed but was running the wrong command!
libtree
not so helpful here unfortunately:
[ocaisa@gpunode1 release]$ ldd deviceQuery
linux-vdso.so.1 (0x00007ffc0d994000)
librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000148061437000)
libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000148061417000)
libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000148061411000)
libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x00001480611fc000)
libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x00001480610bb000)
libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014806109e000)
libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000148060ee3000)
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014806144e000)
[ocaisa@gpunode1 release]$ libtree deviceQuery
deviceQuery
[ocaisa@gpunode1 release]$
haha, but did libtree give you the right results on Gentoo Prefix? Try with -p
and preferably 3.0.0-rc version :) I don't know how Gentoo Prefix manages their default search paths if they put the runtime linker in a prefix too
Sorry, use libtree -vp ...
, as libtree doesn't show c / c++ libs by default to reduce verbosity.
Ok, things are starting to at least look similar now:
[ocaisa@gnode1 release]$ libtree --version
3.0.0-rc7
[ocaisa@gnode1 release]$ libtree -vp ./deviceQuery
./deviceQuery
├── /lib64/librt.so.1 [default path]
│ ├── /lib64/libpthread.so.0 [default path]
│ └── /lib64/libc.so.6 [default path]
├── /lib64/ld-linux-x86-64.so.2 [default path]
├── /lib64/libc.so.6 [default path]
├── /lib64/libgcc_s.so.1 [default path]
├── /lib64/libm.so.6 [default path]
├── /lib64/libstdc++.so.6 [default path]
├── /lib64/libdl.so.2 [default path]
└── /lib64/libpthread.so.0 [default path]
[ocaisa@gnode1 release]$ ldd deviceQuery
linux-vdso.so.1 (0x000014ba62c49000)
librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x000014ba62c2e000)
libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x000014ba62c0e000)
libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x000014ba62c08000)
libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x000014ba629f3000)
libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x000014ba628b2000)
libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014ba62895000)
libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x000014ba626da000)
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014ba62c4b000)
Right. So the prefix is missing. Is /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
a wrapper around the actual runtime linker that sets the gentoo prefix dir as a default library search path?
Our Gentoo Prefix is using it's own glibc with these paths baked in
[ocaisa@gpunode1 release]$ ls -l /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 cvmfs cvmfs 10 Jun 22 13:22 /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 -> ld-2.33.so
I think that answers your question but I'm not an expert here (@bedroge knows a lot more).
Thanks, would be good to know how Gentoo Prefix sets its default library search directories. libtree
currently hard-codes them.
I was thinking to add a --sys-root
flag, which is useful for cross compilation, but it could maybe also help with gentoo prefix.
I'm not the real expert on Gentoo Prefix either (@amadio is one of the Prefix developers and often helping us out, so maybe he can correct me if I'm wrong), but Prefix also uses a custom ld.so.conf
to make sure that stuff is picked up from the alternative sysroot:
$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf
# ld.so.conf autogenerated by env-update; make all changes to
# contents of /etc/env.d directory
include ld.so.conf.d/*.conf
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib
# and for libstdc++:
$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0
Right, so I think I could add Gentoo Prefix support through a flag libtree --sys-root /path/to/gentoo/prefix [args]
and then prepend that to the standard paths <sys-root>/lib
, <sys-root>/etc/ld.so.conf
, ...
Although there could be edge cases where elf files have an absolute path to a lib in their DT_NEEDED, in which case it probably should not prepend the prefix 🤔
Edit: nope, seems like --sys-root would not work, since there are absolute paths including the prefix inside of ld.so.conf already, so it's quite different from chroot-type of environments.
On master you can now do this:
libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...]
The command lddtree
is from app-misc/pax-utils
from Gentoo. I saw libtree posted recently on HN, what a coincidence. In Gentoo Prefix standalone (which EESSI uses), the ELF interpreter is set to something inside the prefix, and using that interpreter to find libraries is important to find the correct answers (using /usr/bin/ldd
will also show bad paths for prefix binaries/libs), as will lddtree
from prefix, unless --prefix
option is used. Only ldd
from prefix shows correct answers by default for me:
$ lddtree $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
libreadline.so.8 => /lib64/libreadline.so.8
libtinfow.so.6 => /lib64/libtinfow.so.6
libtinfo.so.6 => /lib64/libtinfo.so.6
libc.so.6 => /lib64/libc.so.6
$ lddtree --prefix /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64 $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8
libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6
libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6
libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6
$ ldd $(which bash)
linux-vdso.so.1 (0x00007ffd8ebd1000)
libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8 (0x00007ff06a317000)
libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6 (0x00007ff06a2d7000)
libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6 (0x00007ff06a11e000)
libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6 (0x00007ff06a0de000)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x00007ff06a3b7000)
$ /usr/bin/ldd $(which bash)
linux-vdso.so.1 (0x00007ffc722e5000)
libreadline.so.8 => /lib64/libreadline.so.8 (0x00007ff5878f1000)
libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007ff5878b2000)
libc.so.6 => /lib64/libc.so.6 (0x00007ff5876ad000)
libtinfow.so.6 => /lib64/libtinfow.so.6 (0x00007ff58766c000)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007ff587a5f000)
I'm not sure how you can support prefix with libtree, but looking at the implementation of lddtree
(which is a shell or python script) may provide some good insights based on its support for the --prefix
option.
Thanks, I believe /usr/bin/ldd used to show the correct output by simply executing the file with some variable set that hints to the dynamic linker that it should not actually execute. But this was considered a security problem when elf files had a different interpreter. So now ldd is probably just a wrapper that runs ld.so <executable>
explicitly, but then it picks the wrong interpreter :D.
So, my current solution is to simply allow the user to set the path to ld.so.conf, assuming this contains all the paths that would otherwise be default search paths. Let me know if libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...]
works for you.
The flag is now part of https://github.com/haampie/libtree/releases/tag/v3.0.0-rc8
This is looking good
[ocaisa@gnode1 release]$ libtree -vp --ldconf "$EPREFIX/etc/ld.so.conf" deviceQuery
deviceQuery
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 [ld.so.conf]
│ ├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]
│ └── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 [ld.so.conf]
└── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]
Great, then I'll release version 3.0.0 soon ™️