Different values to `ldd` in Gentoo Prefix

Question

Different values to `ldd` in Gentoo Prefix

ocaisa opened this issue 3 years ago · comments

I was trying to use libtree for some testing under Gentoo Prefix. I know we may need to patch CUDA to look in the prefix but I was surprised that ldd and lddtree don't return the same information:

[ocaisa@gpunode1 release]$ ldd ./deviceQuery
        linux-vdso.so.1 (0x00007ffd78ed6000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000153ce5eac000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000153ce5e8c000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000153ce5e86000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x0000153ce5c71000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x0000153ce5b30000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x0000153ce5b13000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000153ce5958000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x0000153ce5ec3000)
[ocaisa@gpunode1 release]$ lddtree ./deviceQuery
deviceQuery => ./deviceQuery (interpreter => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    librt.so.1 => /lib64/librt.so.1
    libpthread.so.0 => /lib64/libpthread.so.0
    libdl.so.2 => /lib64/libdl.so.2
    libstdc++.so.6 => /lib64/libstdc++.so.6
    libm.so.6 => /lib64/libm.so.6
    libgcc_s.so.1 => /lib64/libgcc_s.so.1
    libc.so.6 => /lib64/libc.so.6
    ld-linux-x86-64.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
[ocaisa@gpunode1 release]$ which ldd
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/bin/ldd

Which is correct here? Is it that libtree is using the system linker rather than the one specified in the executable?

Harmen Stoppels · Answer 1 · Tue Dec 14 2021 16:49:59 GMT+0800 (China Standard Time)

lddtree != libtree, seems like you're opening an issue in the wrong repo?

ocaisa · Answer 2 · Tue Dec 14 2021 17:17:05 GMT+0800 (China Standard Time)

Argh, idiot, sorry, I do have libtree installed but was running the wrong command!

ocaisa · Answer 3 · Tue Dec 14 2021 17:23:33 GMT+0800 (China Standard Time)

libtree not so helpful here unfortunately:

[ocaisa@gpunode1 release]$ ldd deviceQuery
        linux-vdso.so.1 (0x00007ffc0d994000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000148061437000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000148061417000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000148061411000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x00001480611fc000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x00001480610bb000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014806109e000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000148060ee3000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014806144e000)
[ocaisa@gpunode1 release]$ libtree deviceQuery
deviceQuery

[ocaisa@gpunode1 release]$

Harmen Stoppels · Answer 4 · Tue Dec 14 2021 17:23:34 GMT+0800 (China Standard Time)

haha, but did libtree give you the right results on Gentoo Prefix? Try with -p and preferably 3.0.0-rc version :) I don't know how Gentoo Prefix manages their default search paths if they put the runtime linker in a prefix too

Harmen Stoppels · Answer 5 · Tue Dec 14 2021 17:30:04 GMT+0800 (China Standard Time)

Sorry, use libtree -vp ..., as libtree doesn't show c / c++ libs by default to reduce verbosity.

ocaisa · Answer 6 · Tue Dec 14 2021 17:47:45 GMT+0800 (China Standard Time)

Ok, things are starting to at least look similar now:

[ocaisa@gnode1 release]$ libtree --version
3.0.0-rc7
[ocaisa@gnode1 release]$ libtree -vp ./deviceQuery
./deviceQuery 
├── /lib64/librt.so.1 [default path]
│   ├── /lib64/libpthread.so.0 [default path]
│   └── /lib64/libc.so.6 [default path]
├── /lib64/ld-linux-x86-64.so.2 [default path]
├── /lib64/libc.so.6 [default path]
├── /lib64/libgcc_s.so.1 [default path]
├── /lib64/libm.so.6 [default path]
├── /lib64/libstdc++.so.6 [default path]
├── /lib64/libdl.so.2 [default path]
└── /lib64/libpthread.so.0 [default path]
[ocaisa@gnode1 release]$ ldd deviceQuery
        linux-vdso.so.1 (0x000014ba62c49000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x000014ba62c2e000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x000014ba62c0e000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x000014ba62c08000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x000014ba629f3000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x000014ba628b2000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014ba62895000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x000014ba626da000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014ba62c4b000)

Harmen Stoppels · Answer 7 · Tue Dec 14 2021 17:57:25 GMT+0800 (China Standard Time)

Right. So the prefix is missing. Is /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 a wrapper around the actual runtime linker that sets the gentoo prefix dir as a default library search path?

ocaisa · Answer 8 · Tue Dec 14 2021 18:02:14 GMT+0800 (China Standard Time)

Our Gentoo Prefix is using it's own glibc with these paths baked in

[ocaisa@gpunode1 release]$ ls -l /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 cvmfs cvmfs 10 Jun 22 13:22 /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 -> ld-2.33.so

I think that answers your question but I'm not an expert here (@bedroge knows a lot more).

Harmen Stoppels · Answer 9 · Tue Dec 14 2021 18:12:25 GMT+0800 (China Standard Time)

Thanks, would be good to know how Gentoo Prefix sets its default library search directories. libtree currently hard-codes them.

I was thinking to add a --sys-root flag, which is useful for cross compilation, but it could maybe also help with gentoo prefix.

Bob Dröge · Answer 10 · Tue Dec 14 2021 18:25:58 GMT+0800 (China Standard Time)

I'm not the real expert on Gentoo Prefix either (@amadio is one of the Prefix developers and often helping us out, so maybe he can correct me if I'm wrong), but Prefix also uses a custom ld.so.conf to make sure that stuff is picked up from the alternative sysroot:

$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf
# ld.so.conf autogenerated by env-update; make all changes to
# contents of /etc/env.d directory
include ld.so.conf.d/*.conf
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib

# and for libstdc++:
$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf 
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0

Harmen Stoppels · Answer 11 · Tue Dec 14 2021 18:30:08 GMT+0800 (China Standard Time)

Right, so I think I could add Gentoo Prefix support through a flag libtree --sys-root /path/to/gentoo/prefix [args] and then prepend that to the standard paths <sys-root>/lib, <sys-root>/etc/ld.so.conf, ...

Although there could be edge cases where elf files have an absolute path to a lib in their DT_NEEDED, in which case it probably should not prepend the prefix 🤔

Edit: nope, seems like --sys-root would not work, since there are absolute paths including the prefix inside of ld.so.conf already, so it's quite different from chroot-type of environments.

Harmen Stoppels · Answer 12 · Tue Dec 14 2021 19:01:13 GMT+0800 (China Standard Time)

On master you can now do this:

libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...]

Guilherme Amadio · Answer 13 · Tue Dec 14 2021 19:07:28 GMT+0800 (China Standard Time)

The command lddtree is from app-misc/pax-utils from Gentoo. I saw libtree posted recently on HN, what a coincidence. In Gentoo Prefix standalone (which EESSI uses), the ELF interpreter is set to something inside the prefix, and using that interpreter to find libraries is important to find the correct answers (using /usr/bin/ldd will also show bad paths for prefix binaries/libs), as will lddtree from prefix, unless --prefix option is used. Only ldd from prefix shows correct answers by default for me:

$ lddtree $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    libreadline.so.8 => /lib64/libreadline.so.8
        libtinfow.so.6 => /lib64/libtinfow.so.6
    libtinfo.so.6 => /lib64/libtinfo.so.6
    libc.so.6 => /lib64/libc.so.6
$ lddtree --prefix /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64 $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8
        libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6
    libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6
    libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6
$ ldd $(which bash)
	linux-vdso.so.1 (0x00007ffd8ebd1000)
	libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8 (0x00007ff06a317000)
	libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6 (0x00007ff06a2d7000)
	libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6 (0x00007ff06a11e000)
	libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6 (0x00007ff06a0de000)
	/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x00007ff06a3b7000)
$ /usr/bin/ldd $(which bash)
	linux-vdso.so.1 (0x00007ffc722e5000)
	libreadline.so.8 => /lib64/libreadline.so.8 (0x00007ff5878f1000)
	libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007ff5878b2000)
	libc.so.6 => /lib64/libc.so.6 (0x00007ff5876ad000)
	libtinfow.so.6 => /lib64/libtinfow.so.6 (0x00007ff58766c000)
	/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007ff587a5f000)

I'm not sure how you can support prefix with libtree, but looking at the implementation of lddtree (which is a shell or python script) may provide some good insights based on its support for the --prefix option.

Harmen Stoppels · Answer 14 · Tue Dec 14 2021 19:12:20 GMT+0800 (China Standard Time)

Thanks, I believe /usr/bin/ldd used to show the correct output by simply executing the file with some variable set that hints to the dynamic linker that it should not actually execute. But this was considered a security problem when elf files had a different interpreter. So now ldd is probably just a wrapper that runs ld.so <executable> explicitly, but then it picks the wrong interpreter :D.

So, my current solution is to simply allow the user to set the path to ld.so.conf, assuming this contains all the paths that would otherwise be default search paths. Let me know if libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...] works for you.

Harmen Stoppels · Answer 15 · Tue Dec 14 2021 19:22:04 GMT+0800 (China Standard Time)

The flag is now part of https://github.com/haampie/libtree/releases/tag/v3.0.0-rc8

ocaisa · Answer 16 · Tue Dec 14 2021 19:36:20 GMT+0800 (China Standard Time)

This is looking good

[ocaisa@gnode1 release]$ libtree -vp --ldconf "$EPREFIX/etc/ld.so.conf" deviceQuery
deviceQuery 
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 [ld.so.conf]
│   ├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]
│   └── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 [ld.so.conf]
└── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]

Harmen Stoppels · Answer 17 · Tue Dec 14 2021 20:06:28 GMT+0800 (China Standard Time)

Great, then I'll release version 3.0.0 soon ™️