libGL not working on non-NixOS (without setting up)

Question

libGL not working on non-NixOS (without setting up)

joepie91 opened this issue 9 years ago · comments

Package: openarena
Error: /nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8/openarena-0.8.8/openarena.x86_64: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory
Platform: openSUSE 13.1, x86_64, XFCE
Version: nix-env (Nix) 1.9

Installation works, running it does not.

sven@linux-etoq:~> nix-env -i openarena
installing ‘openarena-0.8.8’
these paths will be fetched (390.19 MiB download, 426.80 MiB unpacked):
  /nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8
fetching path ‘/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8’...

*** Downloading ‘https://cache.nixos.org/nar/194liswh0vgrj02q7q7ww8igizn45bri6ip4m5wxa47s3ghj65q7.nar.xz’ to ‘/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  390M  100  390M    0     0  7996k      0  0:00:49  0:00:49 --:--:-- 9247k

building path(s) ‘/nix/store/j0lc0hns2kna9sw1g3k46zbazxhsxha0-user-environment’
sven@linux-etoq:~> openarena 
/nix/store/q8smn7k0y349ymfc9mdnja0wa6s79njv-openarena-0.8.8/openarena-0.8.8/openarena.x86_64: error while loading shared libraries: libGL.so.1: cannot open shared object file: No such file or directory

Bjørn Forsman · Answer 1 · Mon Aug 24 2015 13:36:23 GMT+0800 (China Standard Time)

Tricky... I wonder if it can be fixed binding openarena to mesa libGL. Then you'd at least have something that runs, although not necessarily optimally for your system.

On NixOS, programs find the selected/best OpenGL implementation here:

$ echo $LD_LIBRARY_PATH 
/run/opengl-driver/lib:/run/opengl-driver-32/lib

Something similar can of course be done with nixpkgs programs on non-NixOS (manually set $LD_LIBRARY_PATH).

CC @vcunat.

Vladimír Čunát · Answer 2 · Mon Aug 24 2015 14:14:40 GMT+0800 (China Standard Time)

There's a deliberate choice to be impure in this respect, as I view libGL* as a part of your GPU driver. I expect using your system one is better even for apps that don't report an error (most do link against our version from mesa).

If one is able, the safest approach on non-nixos should be to actually create /run/opengl-driver/lib (speaking of x86_64-linux) and put symlinks to your system libGL*. Alternatively one could just set $LD_LIBRARY_PATH to the location, but beware if it contains too many other libs...

Sven Slootweg · Answer 3 · Mon Aug 24 2015 20:31:29 GMT+0800 (China Standard Time)

Hmm. Would this not be a good usecase for parameters (letting you pick between a system libGL* path or a mesa dependency)? Or can those not be specified from the command-line?

(I'm still pretty new to Nix)

Vladimír Čunát · Answer 4 · Mon Aug 24 2015 21:05:05 GMT+0800 (China Standard Time)

You can easily use $LD_LIBRARY_PATH to open any libGL version. I've only seen one person who wanted to use the bundled mesa instead of system libGL (and on NixOS it's often the same one).

Domen Kožar · Answer 5 · Fri Sep 04 2015 16:30:34 GMT+0800 (China Standard Time)

couldn't we require a nixpkgs config setting to be set and wrap all libGL software with it? that way users would get a message how/where to set the setting

Vladimír Čunát · Answer 6 · Fri Sep 04 2015 16:44:57 GMT+0800 (China Standard Time)

@domenkozar: I don't understand what you mean. Note that the problems only happen on non-nixos, and different distros have libGL on different locations, I think.

Domen Kožar · Answer 7 · Fri Sep 04 2015 16:46:19 GMT+0800 (China Standard Time)

I'm saying we could use nixpkgs config to set $LD_LIBRARY_PATH for non-nixos users

Sven Slootweg · Answer 8 · Thu Sep 10 2015 01:18:48 GMT+0800 (China Standard Time)

To a non-technical user, what would the correct (current) procedure look like for installing openarena on a non-NixOS distribution? I'm technical, but I don't know much about compiler toolchains, so an ELI5-style explanation might be the best way to make sense of this.

Sven Slootweg · Answer 9 · Sat Sep 12 2015 04:33:36 GMT+0800 (China Standard Time)

After talking about this with @ttuegel for a while, we've established that the following works:

Install patchelf (nix-env -i patchelf)
Copy the system libGL.so.1 (usually in /usr/lib64) somewhere, eg. ~/.libgl-shim/libGL.so.1.
patchelf --set-rpath /lib:/usr/lib:/lib64:/usr/lib64 ~/.libgl-shim/libGL.so.1 (adjust lib paths if necessary for your distribution)
Replicate other libGL.so.* symlinks that your system has, but in your ~/.libgl-shim, and pointing to your modified copy.
Run openarena as follows: LD_LIBRARY_PATH=~/.libgl-shim openarena

This approach also seems automatable, and should work for other things that use makeWrapper :)

I'm still working on something, but once I've gotten further with that, I'll probably do a proper writeup of this on my blog or something.

Bjørn Forsman · Answer 10 · Sat Sep 12 2015 05:42:13 GMT+0800 (China Standard Time)

What does this give beyond what the simpler LD_LIBRARY_PATH=/usr/lib64 openarena provides (substitute for whatever path libGL.so.1 has on your system)? It seems you still need to provide the custom path to libGL, which cannot be automated.

Sven Slootweg · Answer 11 · Sat Sep 12 2015 05:49:15 GMT+0800 (China Standard Time)

@bjornfor You'd only load the system libGL, and the rest of the dependencies would remain the pure Nix variants.

Bjørn Forsman · Answer 12 · Sat Sep 12 2015 05:51:10 GMT+0800 (China Standard Time)

Good point :-)

Bjørn Forsman · Answer 13 · Sat Sep 12 2015 16:49:21 GMT+0800 (China Standard Time)

What about LD_PRELOAD=/lib/libGL.so.1 openarena then?

Sven Slootweg · Answer 14 · Sat Sep 12 2015 19:03:09 GMT+0800 (China Standard Time)

That wouldn't work because of libGL dependencies:

sven@linux-etoq:~/projects/nixpkgs/test> LD_PRELOAD=/usr/lib64/libGL.so.1 openarena
/nix/store/zmd4jk4db5lgxb8l93mhkvr3x92g2sx2-bash-4.3-p39/bin/bash: error while loading shared libraries: libglapi.so.0: cannot open shared object file: No such file or directory

It'd only look in the Nix store, since Nix uses a custom ld. By elfpatching it, we have a copy of libGL that will also look for its own dependencies in the system-wide library directories, without affecting non-libGL dependencies of the application.

Bjørn Forsman · Answer 15 · Sat Sep 12 2015 19:11:00 GMT+0800 (China Standard Time)

I see. I'm going to stop suggesting alternatives now :-)

Roland Freikamp · Answer 16 · Wed Oct 07 2015 21:00:17 GMT+0800 (China Standard Time)

I tried this according to the instructions of joepie91 above, since I wanted to use OpenSCAD from Nix on Debian Jessie with Intel HD graphics (Haswell). Copying/patching libGL.so.1 was not enough, and glxinfo/OpenSCAD/OpenArena failed because they couldn't find i965_dri.so. Adding i965_dri.so also wasn't enough, because i965_dri.so needed some more libraries to be loaded. After adding these libraries, it worked. :)

The contents of my ~/.nix-libs is now:

libGL.so -> libGL.so.1.2.0
libGL.so.1 -> libGL.so.1.2.0
libGL.so.1.2.0
libpciaccess.so.0
dri/i965_dri.so
# edit: removed libz.so.1

Commands used:

mkdir ~/.nix_libs
rsync -aszvi /usr/lib/x86_64-linux-gnu/libGL.* ~/.nix_libs/
patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0

mkdir ~/.nix_libs/dri
rsync -aszvi /usr/lib/x86_64-linux-gnu/dri/i965_dri.so ~/.nix_libs/dri/
patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/dri/i965_dri.so

cp -pi /nix/store/0fp3kp6kx12wqs593xwk7l6pbq0dkrdp-libpciaccess-0.13.4/lib/libpciaccess.so.0 ~/.nix_libs/

# edit: now works without the following lines
# (for OpenArena additionally:)
# (cp -pi /nix/store/31w31mc8immhpnmxvcl4l0fvc3i5iwh0-zlib-1.2.8/lib/libz.so.1 ~/.nix_libs/)

How to find out which libraries are required in ~/.nix_libs for some program:

Follow joepie91's instructions:

mkdir ~/.nix_libs
rsync -aszvi /usr/lib/x86_64-linux-gnu/libGL.* ~/.nix_libs/
patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0

Run your program (e.g. glxinfo, openscad or openarena) with LD_LIBRARY_PATH:
```
LD_LIBRARY_PATH=~/.nix_libs openscad
```
If the program does not start or OpenGL does not work, search the error-message in the output (e.g.: libGL error: unable to load driver: i965_dri.so)
Run the program again with strace:
```
LD_LIBRARY_PATH=~/.nix_libs strace openscad 2>&1 | less -S
```
and search the output for the error-message and the missing library or the fatal ENOENT.
Make sure that this missing library can be found by copying it into ~/.nix_libs (or a subdirectory) and patchelf-ing it (see step 1).
Go to step 2.

By the way, glxinfo still aborts with glxinfo: : ATUSHûHìH-Ó : Error 18644416 (same for glxgears), but OpenSCAD and OpenArena work.

Vladimír Čunát · Answer 17 · Wed Oct 07 2015 22:15:07 GMT+0800 (China Standard Time)

LD_LIBRARY_PATH=~/.nix_libs is slightly dangerous with having standard libs in there (libz in particular). It means that all nix-installed stuff will use that libz instead of the one they were compiled against. But if the list is very small, it might cause no problems (and it's rather easy to set up).
DRI drivers are located by libGL – I'm not sure how distros typically configure libGL's search path; it might even differ.
For inspecting (broken dynamic) linkage it seems easiest to run ldd on the executables or libraries, but it won't show _dlopen_ed libs and some programs do use that to open libGL.
There are also some other libs taken from that path, e.g. libvdpau_*.so. (I don't remember how libva works ATM.)

Roland Freikamp · Answer 18 · Thu Oct 08 2015 00:01:20 GMT+0800 (China Standard Time)

LD_LIBRARY_PATH=~/.nix_libs is slightly dangerous with having standard libs in there (libz in particular). It means that all nix-installed stuff will use that libz instead of the one they were compiled against.

Since I only call some OpenGL-needing-programs with LD_LIBRARY_PATH=~/.nix_libs, only those use that libz.
I would of course like to remove the standard libs from my ~/.nix_libs, but I don't know how to do this without breaking the OpenGL-needing-programs, since the libs are only searched in .nix_libs and some "wrong" nix-store-paths:

open("/nix/store/nkp9sq3jj09wjxmq4h5zyaa0spc6nsrs-SDL-1.2.15/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/ikgb8896hqcy7pz7n35v1y6qcifim9bi-libogg-1.3.2/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/w0p7bn2rzyd4q8haggr5z7pyqh9kmmgk-libvorbis-1.3.5/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/user/.nix_libs/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/nix/store/hd6km3hscbgl2yw8nx7lr5z9s8h89p04-glibc-2.21/lib/libpciaccess.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

(But I've removed libz now, since strangely it now works without ~/.nix_libs/libz.so.1 after uninstalling and reinstalling...)

DRI drivers are located by libGL

Unfortunately, the i965_dri.so is only searched in the wrong places, so I had to copy it to .nix_libs/dri/tls/i965_dri.so:

open("/home/user/.nix_libs/dri/tls/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/user/.nix_libs/dri/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/tls/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/usr/lib/dri/i965_dri.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

For inspecting (broken dynamic) linkage it seems easiest to run ldd on the executables or libraries

Yes, but neither ldd nor LD_DEBUG helped here, since they did not show the missing libraries.

Vladimír Čunát · Answer 19 · Thu Oct 08 2015 00:41:06 GMT+0800 (China Standard Time)

Unfortunately, the i965_dri.so is only searched in the wrong places, so I had to copy it to .nix_libs/dri/tls/i965_dri.so:

That part is a very good solution, I think, except symlink should be better than copy (due to your OS updates).

Yes, but neither ldd nor LD_DEBUG helped here, since they did not show the missing libraries.

Hmm, then I suppose these are also cases using dlopen, unfortunately. I know nothing better than strace for that, which is what you used.

Matthias C. M. Troffaes · Answer 20 · Tue Dec 08 2015 23:59:48 GMT+0800 (China Standard Time)

I just ran into a similar issue trying to get nix's blender running on Fedora: blender wouldn't find i965_dri.so. Here are the steps that I had to do to make this work - notably I also had to set LIBGL_DRIVERS_PATH in addition to LD_LIBRARY_PATH (hence why I'm posting this here - this hasn't been mentioned above yet):

Create a folder for any additionally required OpenGL drivers/libraries.

Run

LIBGL_DEBUG=verbose LD_LIBRARY_PATH=/path/to/drivers LIBGL_DRIVERS_PATH=/path/to/drivers blender

If this works, great! You can drop LIBGL_DEBUG=verbose in future invokations.

If it doesn't work due to a missing file, create a symlink to the missing file from your driver folder and go back to step 2.

LD_DEBUG=libs may help too in step 2 in case you don't get enough output from LIBGL_DEBUG=verbose - but I didn't have to use it.

davidak · Answer 21 · Thu Dec 10 2015 21:04:42 GMT+0800 (China Standard Time)

i can reproduce it on elementary OS Freya (based on Ubuntu 14.04 LTS).

i copied the systems libGL.so and patched it, then openarena works.

# apt-get install libgl1-mesa-dev
$ cp /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1.2.0 ~/.nix_libs/
$ patchelf --set-rpath /lib:/usr/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu ~/.nix_libs/libGL.so.1.2.0
$ cd ~/.nix_libs
$ mv libGL.so.1.2.0 libGL.so
$ LD_LIBRARY_PATH=~/.nix_libs openarena

Vladimír Čunát · Answer 22 · Fri Dec 18 2015 22:04:15 GMT+0800 (China Standard Time)

It still seems rather inconvenient that one needs to do update libGL by hand (after the host OS updates).

Johan Boule · Answer 23 · Tue Jan 12 2016 03:24:39 GMT+0800 (China Standard Time)

I've had success using the following wrapper script :

#! /bin/sh

set -x &&

sys=/usr/lib/x86_64-linux-gnu &&

LIBGL_DEBUG=verbose \
LIBGL_DRIVERS_PATH=$sys/dri \
LD_PRELOAD=\
$sys/libdrm.so:\
$sys/libdrm_intel.so:\
$sys/libdrm_nouveau.so:\
$sys/libdrm_radeon.so:\
$sys/libpciaccess.so.0:\
$sys/libz.so:\
$sys/libstdc++.so.6 \
exec "$@"

davidak · Answer 24 · Tue Jan 12 2016 08:06:03 GMT+0800 (China Standard Time)

on ubuntu i would have to add a lot more librarys. it is easier to add a path like

$ LIBGL_DEBUG=verbose LIBGL_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu kdenlive

With openarena i get a Segmentation fault, with kde5.kdenlive this error:

$ LIBGL_DEBUG=verbose LIBGL_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri LD_LIBRARY_PATH=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu kdenlive
kdenlive: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /nix/store/2hp0a5arv6bi8zii79xhy23nagh7470s-qtbase-5.4.2/lib/libQt5Core.so.5)
kdenlive: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /nix/store/gfzkhby7nlz97r5g0ny8qwmlrjnsjm0y-icu4c-55.1/lib/libicui18n.so.55)
kdenlive: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /nix/store/gfzkhby7nlz97r5g0ny8qwmlrjnsjm0y-icu4c-55.1/lib/libicuuc.so.55)

this happens because the package needs GCC >=4.9 but Ubuntu 14.04 uses 4.8

so i installed it using nix with nix-env -iA nixos.pkgs.gcc, add the library to the path and i can finally start it, but it also crashes with a Segmentation fault :(

$ LIBGL_DEBUG=verbose LIBGL_DRIVERS_PATH=/usr/lib/x86_64-linux-gnu/dri LD_LIBRARY_PATH=/nix/store/0lx4bbsik6xmi8qs0x0g76jha59m2m2n-user-environment/lib/:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu kdenlive
Couldn't start kglobalaccel from org.kde.kglobalaccel.service: QDBusError("org.freedesktop.DBus.Error.ServiceUnknown", "The name org.kde.kglobalaccel was not provided by any .service files")
mlt_repository_init: failed to dlopen /nix/store/h8jq0q7gyyh1x2hxssq6b9135b6kdjj3-mlt-0.9.6/lib/mlt/libmltavformat.so
  (/nix/store/mhzz265j047cxsbmfrkci06j6wq77j47-gnutls-3.4.5/lib/libgnutls.so.30: symbol asn1_decode_simple_ber, version LIBTASN1_0_3 not defined in file libtasn1.so.6 with link time reference)
QFSFileEngine::open: No file name specified
QFSFileEngine::open: No file name specified
libGL: screen 0 does not appear to be DRI3 capable
libGL: pci id for fd 10: 1002:95c5, driver r600
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/tls/r600_dri.so
libGL: OpenDriver: trying /usr/lib/x86_64-linux-gnu/dri/r600_dri.so
libGL: Can't open configuration file /home/davidak/.drirc: No such file or directory.
libGL: Can't open configuration file /home/davidak/.drirc: No such file or directory.
libGL: Using DRI2 for screen 0
Icon theme "oxygen" not found.
kf5.kiconthemes: Couldn't find current icon theme, falling back to default.
Icon theme "oxygen" not found.
Error: standard icon theme "oxygen" not found!
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
QWidget::setMaximumSize: (/QToolButton) Negative sizes (2,-2) are not possible
kf5.kxmlgui: Registering action  "edit_select_all"  under new name  "select_all_tracks"
libDeckLinkAPI.so: cannot open shared object file: No such file or directory
QCoreApplication::postEvent: Unexpected null receiver
QFSFileEngine::open: No file name specified
ui/ui_standards.rc not found in ("/home/davidak/.config", "/etc/xdg/xdg-pantheon", "/etc/xdg")
cannot find .rc file "kdenliveui.rc" for component "kdenlive"
Speicherzugriffsfehler (Speicherabzug geschrieben)

the end again with strace:

stat("/usr/share/kdenlive/kdenliveui.rc", 0x7ffe41f78420) = -1 ENOENT (No such file or directory)
write(2, "cannot find .rc file \"kdenliveui"..., 62cannot find .rc file "kdenliveui.rc" for component "kdenlive"
) = 62
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\4\0\2\0\340\0\300\2\4\0\2\0\335\0\300\2", 16}], 1) = 16
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x2aabc68, FUTEX_WAIT_PRIVATE, 2, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 0
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"N\0\4\0\337\0\300\2\6\0\300\2\17\2\0\0\1\30\v\0\343\0\300\2\6\0\300\2\0\0\0\0"..., 60}, {NULL, 0}, {"", 0}], 3) = 60
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\2\0\7\0\343\0\300\2\1\16\0\0\0\0\0\0\0\0\0\0\0\0\0\0\177\240b\0\22\0\30\0"..., 400}], 1) = 400
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\203.\7\0\343\0\300\2\2\0\300\2\10\0\1\0`\0\0\0\4\0\1\0`\0\0\0\203.\5\0"..., 48}, {NULL, 0}, {"", 0}], 3) = 48
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\2\0\5\0\343\0\300\2\0\n\0\0\0\0\0\0\177\240b\0\22\0\7\0\343\0\300\2F\1\0\0"..., 116}], 1) = 116
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7ffe41f77d20, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x7ffe41f77d24, FUTEX_WAIT_PRIVATE, 1, NULL) = -1 EAGAIN (Resource temporarily unavailable)
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 0
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\22\0\17\0\343\0\300\2#\0\0\0#\0\0\0 \0\0\0\t\0\0\0\3\0\0\0\1\0\0\0"..., 92}], 1) = 92
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"+.\1\0", 4}, {NULL, 0}, {"", 0}], 3) = 4
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
getcwd("/home/davidak", 4098)           = 14
getcwd("/home/davidak", 4098)           = 14
getcwd("/home/davidak", 4098)           = 14
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
poll([{fd=3, events=POLLIN|POLLOUT}], 1, 4294967295) = 1 ([{fd=3, revents=POLLOUT}])
writev(3, [{"\22\0\7\0\343\0\300\2l\1\0\0\4\0\0\0 \0\0\0\1\0\0\0\5\0\0\0\22\0\30\0"..., 240}], 1) = 240
futex(0x2aabc68, FUTEX_WAKE_PRIVATE, 1) = 1
write(5, "\1\0\0\0\0\0\0\0", 8)         = 8
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} ---
+++ killed by SIGSEGV (core dumped) +++
Speicherzugriffsfehler (Speicherabzug geschrieben)

Vladimír Čunát · Answer 25 · Tue Jan 12 2016 15:19:26 GMT+0800 (China Standard Time)

@davidak: that's all because setting LD_LIBRARY_PATH overrides the libs for all binaries, even nix-installed ones. Note the broken dependency chain:

/usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /nix/store/2hp0a5arv6bi8zii79xhy23nagh7470s-qtbase-5.4.2/lib/libQt5Core.so.5)

Tyson Whitehead · Answer 26 · Thu Feb 18 2016 02:32:59 GMT+0800 (China Standard Time)

I may be missing something here as I'm not actually running NixOS, but, I'm not sure I follow why mesa has to be patched to look for it's dri libraries in /run/opengl-driver{,-32}/lib.

That is, say it was't patched and looked in the $drivers output instead. Wouldn't this still work fine so long as the appropriate libGL is symlinked into the /run/... directory that is in LD_LIBRARY_PATH.

If it is the mesa one, then it would find its drivers in the $drivers output and if it was some proprietary one (NVIDIA/AMD) then it wouldn't care about the mesa dri drivers.

Vladimír Čunát · Answer 27 · Thu Feb 18 2016 02:57:53 GMT+0800 (China Standard Time)

We plan to get rid of that $LD_LIBRARY_PATH on NixOS, or at least reduce its usage, as it's an error-prone technique.

davidak · Answer 28 · Thu Feb 18 2016 16:52:09 GMT+0800 (China Standard Time)

@vcunat will that solve also issues with programs installed with Nix on non-nixos machines?

Vladimír Čunát · Answer 29 · Thu Feb 18 2016 17:03:27 GMT+0800 (China Standard Time)

No, I don't think it would really help on non-NixOS.

Peter Simons · Answer 30 · Thu Feb 18 2016 17:13:45 GMT+0800 (China Standard Time)

The wrapper script posted in #9415 (comment) is probably the best solution for users of Nix on host systems other than NixOS. You'll have to adapt the paths and the selection of libraries for your local machine, of course, but I've run Blender, mplayer, and mpv successfully on openSUSE with that approach.

Vladimír Čunát · Answer 31 · Thu Feb 18 2016 17:23:31 GMT+0800 (China Standard Time)

Perhaps, though preloading general libs like libz and libstdc++ is also asking for trouble. The problem is that the environment variables don't affect just libGL but everything.

davidak · Answer 32 · Thu Feb 18 2016 17:29:02 GMT+0800 (China Standard Time)

It is also no intuitive solution for "ordinary users" of Nix.
You should be able to just install and use a program. (if possible somehow)

Peter Simons · Answer 33 · Thu Feb 18 2016 20:19:21 GMT+0800 (China Standard Time)

Well, we all agree that we don't like the LD_PRELOAD solution. However, what do you suggest we should do instead? Stating how things "should" be is one thing. Suggesting a viable path to that state is another.

davidak · Answer 34 · Thu Feb 18 2016 22:30:36 GMT+0800 (China Standard Time)

@peti Okay. Unfortunately i have no better solution. :/

Tyson Whitehead · Answer 35 · Fri Feb 19 2016 00:08:34 GMT+0800 (China Standard Time)

I hope we can come up with something that also works outside of NixOS.

It would, in particular, be nice to avoid paths like /run/opengl-driver{,-32}/lib showing up in the Nix expressions as changing them to something more appropriate for non-NixOS environments means you can't use the binary cache. To that end I would suggest either

stick it under a nix prefixed directory (e.g., /run/nix/... or /nix/run/...) as this is less likely to be objectionable on a non-NixOS system, or
parameterize it in the expressions and have hydra build a NixOS version using /run/... and something else more friendly for non-NixOS such as /nix/run/....

With regard the LD_LIBRARY_PATH, possibly we should be doing the indirection through something like the xorg_sys_opengl package. That is, a package that provides symlinks from the store directly to outside libraries or (better) symlinks from the store to a second level of indirection symlinks like /run/opengl-driver{,-32}/lib that then symlink to the final libraries.

Vladimír Čunát · Answer 36 · Fri Feb 19 2016 15:14:55 GMT+0800 (China Standard Time)

Why do you think /nix/* is more suitable than /run/*? When I read FHS item on wiki, /run seems to fit very well:

Run-time variable data: Information about the running system since last boot, e.g., currently logged-in users and running daemons.

Indirection through a package may seem nice at first, but note that it would have to be an inherently impure package, as the symlinks have to lead to different locations based on configuration of your OS. For example on an Ubuntu instance I see it's just in /usr/lib/nvidia-352/ which will change even on major driver update (!).

davidak · Answer 37 · Fri Feb 19 2016 16:09:47 GMT+0800 (China Standard Time)

Maybe a Script or systemd timer that is run at startup and collects all needed paths and symlinks them?

Tyson Whitehead · Answer 38 · Sat Feb 20 2016 02:15:11 GMT+0800 (China Standard Time)

@vcunat I was thinking mostly of permissions and containment on non-NixOS systems

the user has permissions required to setup things under /nix/*
nix is fully contained under /nix* so it can't conflict with anything else and removing /nix* removes it

Vladimír Čunát · Answer 39 · Sat Feb 20 2016 15:42:18 GMT+0800 (China Standard Time)

I see. Though, on NixOS it seems strange to put it in there, and to share binaries we probably want to share the paths. I don't know.

Ole Jørgen Brønner · Answer 40 · Wed Mar 09 2016 22:44:35 GMT+0800 (China Standard Time)

Anyone had any luck with proprietary nvidia driver?

Ellie Hermaszewska · Answer 41 · Thu Apr 07 2016 17:20:27 GMT+0800 (China Standard Time)

Has anyone given any thought to overriding mesa (or other packages depended upon for libGL and friends) in the user's local config with a version matching what the user has on their system?
I'm sure it's not quite as simple as this, but it does seem to fit quite neatly into the existing packaging framework.

Frederik Rietdijk · Answer 42 · Tue Jul 05 2016 04:38:52 GMT+0800 (China Standard Time)

How about we add wrappers for the different distro's to Nixpkgs and have a function libGLWrap which creates a derivation that wraps the original derivation and where the specific wrapper is chosen with a nixpkgs.config option?

Ashley Gillman · Answer 43 · Thu Jan 19 2017 12:19:56 GMT+0800 (China Standard Time)

I am trying to install mrtrix3's mrview on Ubuntu (a QT app) using #9415 (comment), does it make sense for me to have to go so far as to including X libraries?

I have managed to get the executable to load using the following, although it does not actually work apart from some of the toolbar.

#! /bin/sh

set -x &&

sys=/lib/x86_64-linux-gnu &&
usr=/usr/lib/x86_64-linux-gnu &&
nvi=/usr/lib/nvidia-367 &&

LIBGL_DEBUG=verbose \
LD_PRELOAD=$usr/libdrm.so:\
$usr/libX11.so.6:\
$usr/libXext.so.6:\
$usr/libxcb.so.1:\
$usr/libXau.so.6:\
$usr/libXdmcp.so.6:\
$nvi/libGL.so:\
$nvi/libGLX.so.0:\
$nvi/libGLdispatch.so.0 \
"$@"

Kosyrev Serge · Answer 44 · Mon Mar 27 2017 09:18:56 GMT+0800 (China Standard Time)

https://github.com/deepfire/nix-install-vendor-gl is an attempt at automated, clean solution to this problem.

Currently nVidia-only -- AMD and Intel contributions welcome!

Btw, the original concept on how to solve this is due to @cleverca22, who first helped me to overcome the GL-no-comprendo issue on SteamOS.

Alex Ozer · Answer 45 · Mon Apr 24 2017 08:23:35 GMT+0800 (China Standard Time)

Any updates / progress?

Alex Ozer · Answer 46 · Sun Apr 30 2017 12:37:08 GMT+0800 (China Standard Time)

@deepfire does your solution support intel and amd cards yet?

Kosyrev Serge · Answer 47 · Mon May 01 2017 04:48:11 GMT+0800 (China Standard Time)

@alexozer, no, unfortunately not -- I've had very little free time lately..

Hoàng Xuân Phú · Answer 48 · Sun Jun 04 2017 14:26:59 GMT+0800 (China Standard Time)

I have a script that makes Intel work in Ubuntu. I spent some time trying to fit it into nix-install-vendor-gl but the task seems overly complex for me so I'm giving up for now. Feel free to give it a try.

Niklas Hambüchen · Answer 49 · Sun Jun 18 2017 08:47:26 GMT+0800 (China Standard Time)

Anyone had any luck with proprietary nvidia driver?

@olejorgenb I use LD_LIBRARY_PATH=/usr/lib/nvidia-375 myprogram to get OpenGL programs inside nix-shell on Ubuntu to work on NVIDIA.

Ellie Hermaszewska · Answer 50 · Mon Jun 19 2017 01:50:44 GMT+0800 (China Standard Time)

FWIW I have this sytemd unit on ubuntu, horribly hardcoded to a specific driver version...

[Unit]
Description="Link OpenGL driver for nix programs"

[Service]
Type=oneshot
ExecStart=/bin/mkdir -p /run/opengl-driver/lib
ExecStart=/bin/sh -c 'ln -s /usr/lib/x86_64-linux-gnu/libcuda.so* /usr/lib/x86_64-linux-gnu/libXext.so* /usr/lib/nvidia-375/* /run/opengl-driver/lib/'
TimeoutSec=0
StandardInput=tty
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Niklas Hambüchen · Answer 51 · Mon Jun 19 2017 04:13:40 GMT+0800 (China Standard Time)

@expipiplus1 Note you can use ldconfig to get the path of the nvidia driver, e.g.

$ ldconfig -f /etc/ld.so.conf -C /etc/ld.so.cache -p | grep libGL.so
	libGL.so.1 (libc6,x86-64) => /usr/lib/nvidia-375/libGL.so.1
	libGL.so.1 (libc6) => /usr/lib32/nvidia-375/libGL.so.1
	libGL.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGL.so
	libGL.so (libc6,x86-64) => /usr/lib/nvidia-375/libGL.so
	libGL.so (libc6) => /usr/lib32/nvidia-375/libGL.so

Carbon based Guibou · Answer 52 · Tue Jul 25 2017 23:15:17 GMT+0800 (China Standard Time)

Do you know something I can do, as an application developer, to ensure my application works on non-NixOS? I cannot ask my client to manually patch their system, however I can do anything ugly in my code to make it work.

Do you think there is something to do with libglvnd? As far as I understood it, a package can be build with libglvnd as a pure dependency and only depend on it. At runtime, libglvnd will search for a suitable driver (I don't know how).

(This issue is really a pain, this is the only reason we still use our ten year old ubuntu machine to build ;)

Niklas Hambüchen · Answer 53 · Wed Jul 26 2017 22:56:14 GMT+0800 (China Standard Time)

@guibou I don't know libglvnd, but you can certainly change your application (technically probably a wrapper script you have around it) to call ldconfig at startup-time, and parse its output to automatically determine which libraries you should LD_PRELOAD.

If you do that, please share your work here!

don bright · Answer 54 · Wed Aug 16 2017 14:17:33 GMT+0800 (China Standard Time)

the whole '/run' thing invalidates the idea that "to uninstall nix, just rm -rf /nix"

and i really just need a software GL driver working... im running on VMs. seems like this should be do-able without interacting at all with system libraries. I tried export LIBGL_ALWAYS_SOFTWARE=1 and it didn't fix anything

Vladimír Čunát · Answer 55 · Sun Aug 20 2017 04:18:16 GMT+0800 (China Standard Time)

Different hardware is best used with different drivers, especially some VMs have their own libGL implementations that utilizes host's acceleration IIRC. (I'd personally use a NixOS VM, but you probably have reasons not to.)

don bright · Answer 56 · Mon Aug 21 2017 10:28:20 GMT+0800 (China Standard Time)

@vcunat GL is not just used for interactive graphics, it is used to generate image files and also for regression testing of programs. VMs are often used for testing, and the hardware GL driver is irrelevant in such cases because you want to test the program itself without dealing with hardware, using software-only-rendering for GL should not involve any hardware-specific gpu drivers of any kind on any system. This is also good for virtual headless X systems like Xvfb or Xvnc when run on remote servers.

(deleted my prev comment about mmap EPERM when copying files to /run, dont copy files to /run)

update
i just noticed that

/run requires root access

/run is not persistent across reboots https://wiki.debian.org/ReleaseGoals/RunDirectory

Tuomas Tynkkynen · Answer 57 · Mon Aug 21 2017 21:54:25 GMT+0800 (China Standard Time)

Setting:

LIBGL_DRIVERS_PATH=$(nix-build --no-out-link -A mesa_drivers)/lib/dri
LD_LIBRARY_PATH=$(nix-build --no-out-link -A mesa_noglu)/lib

should work (assuming no proprietary GPU drivers) without touching /run.

Carbon based Guibou · Answer 58 · Thu Aug 24 2017 17:34:25 GMT+0800 (China Standard Time)

I'm trying to understand the issue. OpenGL is, by definition, impure. You need to link with the system provided libGL.so to get something working with hardware acceleration.

We usually link with the libGL provided in nixpkgs in the package mesa-noglu, and that one fails at loading the necessary shared libraries. Most of the "hackish" solutions involves copying the shared libraries in /run/... plus changing the runpath of them and all of their recursive dependencies. This is complicated, system dependent and highly intrusive.

Another solution proposed in nix-install-vendor-gl.sh is to install in the nix store the same driver and library as those provided by the system. This seems a good idea, however it is a pain to work for any system, any driver version, ... As far as I know, it only support nvidia main line of driver, on ubuntu.

However, it is possible to add to our executable a runpath (in last position) which is the one of the host system (usually /usr/lib). This way, the program will find the system libGL. However it stills fail to link because the system libGL have an empty runpath: it depends on the system dynamic loader builtin list. The nix dynamic loader does not comes with this list.

What will happen if we ask the nix dynamic loader to respect the system provided library configuration AFTER the runpath provided inside the executable / shared libraries? I have the feeling that it will not break anything nix related, because a correctly built nix package should find its dynamic library in the directory provided as runpath, however it may fix most of the issues with OpenGL, and, I hope, may also work with weird configurations such as primus.

Did I miss something?

Side note: the case where you are sure you want the software mesa GL can be handled differently, by, for example, providing a "mesa-pure" package.

Carbon based Guibou · Answer 59 · Sat Sep 16 2017 19:58:02 GMT+0800 (China Standard Time)

I just experimented with that https://github.com/guibou/nixGL solution.

It works out of the box for me, and I find the idea elegant (disclaimer: that's my idea ;) and it should be really easy to adapt to any system. It is just a wrapper script, and it does not change anything on the host system. The wrapper script can be installed with nix.

This solves most of my issues with OpenGL on non nix system (with the drawback of needing to call the binary through a wrapper script).

Vladimír Čunát · Answer 60 · Sat Sep 16 2017 20:15:51 GMT+0800 (China Standard Time)

That could work well enough. I believe it might do a bit unexpected things when running sub-processes, due to the fact that LD_LIBRARY_PATH is just inherited. Also running non-ELF executables probably won't work with this script, e.g. those produced via makeWrapper.

Carbon based Guibou · Answer 61 · Sat Sep 16 2017 20:38:39 GMT+0800 (China Standard Time)

@vcunat I'm now convinced that the only solution to fix this issue permanently and in robust way is a patch in the dynamic-loader. When the dynamic loader looks for libGL.so, instead of following the rpath of the current executable, it will follows the rules of the current operating system (using their dynamic loader settings) and will do this recursively.

Bjørn Forsman · Answer 62 · Sat Sep 16 2017 20:45:09 GMT+0800 (China Standard Time)

@guibou: Interesting!

Vladimír Čunát · Answer 63 · Sat Sep 16 2017 20:48:12 GMT+0800 (China Standard Time)

There's a more difficult problem in there, similarly to NixOS – various dependencies of libGL. Unless we want some real black magic, you will only have one version of (say) libc loaded at once in a process – either ours or the OS one has to win. Well, libc itself is probably the less risky one in this, but some implementations depend on more fragile libs (we had this libwayland issue).

Tuomas Tynkkynen · Answer 64 · Sat Sep 16 2017 20:56:28 GMT+0800 (China Standard Time)

Yes. Hopefully SomeoneElse(tm) gets the black magic working: https://sourceware.org/ml/libc-help/2017-05/msg00016.html

Tyson Whitehead · Answer 65 · Sat Oct 14 2017 00:00:16 GMT+0800 (China Standard Time)

I didn't have any luck trying to get my Debian MESA OpenGL library to work with Nix executables. What did work though was just using the Nix MESA OpenGL with the Nix executables.

sudo bash -c "$(declare -p); nix-build -A 'mesa_noglu.drivers' $HOME/.nix-defexpr/channels/nixpkgs -o /run/opengl; mv -T /run/opengl-driver{s,}"

This just creates a symlink from /run/opengl-driver to the drivers output of the Nix mesa_noglu package and registers it with the garbage collector. nix-build suffixes the output with -drivers, so we have to change that to -driver (this was missing as noted by @woffs later in the first version)..

The $(declare -p) bit imports the users environment into the superuser's environment. Useful as likely Nix is installed in single-user mode and the superuser won't be setup correctly to use it.

Kosyrev Serge · Answer 66 · Sat Oct 14 2017 00:11:03 GMT+0800 (China Standard Time)

@twhitehead, cool! What GPU stack do you have, Intel?

Frank Doepper · Answer 67 · Sat Oct 14 2017 00:37:48 GMT+0800 (China Standard Time)

Great, works, after doing
ln -s opengl-drivers /run/opengl-driver
:-)

Tyson Whitehead · Answer 68 · Sat Oct 14 2017 01:31:26 GMT+0800 (China Standard Time)

@deepfire AMD. It's an Asus Radeon R9 270 IIRC. Bought it a couple of years ago specifically so I didn't have to mess around with any closed source pain and the large slow fans made it super quiet.

Tuomas Tynkkynen · Answer 69 · Tue Oct 31 2017 19:39:58 GMT+0800 (China Standard Time)

I have been trying to get libcapsule working: https://github.com/dezgeg/libcapsule https://github.com/dezgeg/nixpkgs/tree/libcapsule. So far it compiles a libGL stub successfully but glxinfo crashes somewhere in the dynamic linker during the very first call it does (glXChooseVisual)...

#0  0x00007ffff7debea0 in add_to_global () from /nix/store/yydnhs7migvlbl48wpsxan1yvq2icbr9-glibc-2.25-49/lib/ld-linux-x86-64.so.2
#1  0x00007ffff7decec4 in dl_open_worker () from /nix/store/yydnhs7migvlbl48wpsxan1yvq2icbr9-glibc-2.25-49/lib/ld-linux-x86-64.so.2
#2  0x00007ffff7598001 in _dl_catch_error () from /nix/store/yydnhs7migvlbl48wpsxan1yvq2icbr9-glibc-2.25-49/lib/libc.so.6
#3  0x00007ffff7dec107 in _dl_open () from /nix/store/yydnhs7migvlbl48wpsxan1yvq2icbr9-glibc-2.25-49/lib/ld-linux-x86-64.so.2
#4  0x00007ffff4fb8f66 in dlopen_doit ()
#5  0x00007ffff5d08001 in _dl_catch_error ()
#6  0x00007ffff4fb9569 in _dlerror_run ()
#7  0x00007ffff4fb8ff1 in dlopen@@GLIBC_2.2.5 ()
#8  0x00007ffff2e68778 in driOpenDriver ()
#9  0x00007ffff2e6f6bb in dri3_create_screen ()
#10 0x00007ffff2e43709 in __glXInitialize ()
#11 0x00007ffff2e3f91b in GetGLXPrivScreenConfig.part.2 ()
#12 0x00007ffff2e3fa83 in glXChooseVisual ()
#13 0x0000000000404148 in mesa_hack (dpy=0x7ffff7eed010, scrnum=0) at glxinfo.c:1168
#14 0x0000000000404417 in main (argc=1, argv=0x7fffffffb2e8) at glxinfo.c:1257

Vladimír Čunát · Answer 70 · Thu Nov 02 2017 02:58:38 GMT+0800 (China Standard Time)

Sounds really interesting/promising. Perhaps we should try asking the author.

don bright · Answer 71 · Tue Dec 12 2017 06:10:13 GMT+0800 (China Standard Time)

inspired by all the above..... i was experimenting with a kludge for OpenSCAD that uses some.. uhm.. .unusual facts about dynamic linkers and directory searching of dlopen() on linux and patchelf. I have tested about a dozen different linux distros on several different x86 machines, including amd hardware, nvidia hardware, intel 915 and 965 hardware, with open source drivers, and nvidia's closed source driver. and the software driver.

https://github.com/openscad/openscad/tree/nixbuild

The basic theory is that you isolate the various dependency library .so files into multiple subdirectories, and mess around with the RPATHs and other unusual features of ELF on linux. In this way you can manipulate not only your main program into loading a specific libGL.so, but you can also manipulate libGL.so into loading specific driver .so files, and also manipulate those driver .so files into loading specific versions of libraries that they need. So you can have two different versions of libwhatever.so, in the overall dependency tree of your main program, and it turns out to be OK.

The duplicate .so files aren't technically linked "directly" into your program... they are loaded individually by the shared object loader at runtime - combining this fact with the hierarchically isolated way that ELF dynamic object files, including libGL.so, load and calls its dependencies at runtime, and our RPATH manipulation, it means that you can have somefunction() inside libwhatever.so that is from Nvidia's proprietary bundle of files, then also have somefunction() inside libwhatever.so from the system and they wont crash or smash each other. This also works with stuff like QT which dynamically loads libGL.so.

The basic theory is here in a tiny program that i used to test this hypothesis:

https://github.com/donbright/mrpblt

I am like 90% sure this is correct? It seemed to work with OpenSCAD ok on, as i said, about a dozen different linux versions, on four machines, and I was deep into testing but it kind of overwhelmed me, the complexity of figuring out the simplest way to organize the build process to where it would work both with Nvidia proprietary drivers and open source drivers. It winds up copying a huge tree of .so files into a subdirectory of the build, inspired by how the __qt5_nix__ subdirectory magically comes into existence for qmake qt5 programs under nix shell. It also requires modifying the generated binary be modified so it only looks under those subdirs for certain things.

Also reading about why Nvidia has proprietary drivers, and how the PC graphics card industry works, was kind of depressing. The amount of time any graphics project spends working around OpenGL issues might be better spent preparing for Vulkan future: https://www.gamedev.net/forums/topic/666419-what-are-your-opinions-on-dx12vulkanmantle/?tab=comments#comment-5215019

Carbon based Guibou · Answer 72 · Sun Feb 18 2018 06:12:30 GMT+0800 (China Standard Time)

Hello.

I did an update of nixGL: https://github.com/guibou/nixGL since I have now a better understanding of the problem. The previous limitations I had with my previous approach are now removed.

Kosyrev Serge · Answer 73 · Sun Feb 18 2018 06:51:39 GMT+0800 (China Standard Time)

@guibou, how does nixGLNvidia address the case when the Nixpkgs-provided nVidia client-side component version doesn't match the nVidia host side? In my experience it's often a crash, which is why nix-install-vendor-gl goes to the pains of version detection etc..

Carbon based Guibou · Answer 74 · Sun Feb 18 2018 19:02:50 GMT+0800 (China Standard Time)

@deepfire as you said it may (will) crash. My solution is now really close to yours. I wanted to contribute in yours instead but I was blocked by my poor shell skills ;)

(Edit: nixGL can now use a specific version, manually specified by the user during nix-build)

cx405 · Answer 75 · Thu Mar 15 2018 07:12:58 GMT+0800 (China Standard Time)

@guibou Hi! I have AMD hardware (R9 280 and also mobility hd4650) and willing to test any task to help out with the project. I am a bit of new though, so sorry upfront, if I'll misunderstand anything.

I noticed that both urban-terror and openarena (nix packages) do not run on nix, causing screen to go blank, and I also have some proprietary titles for what's worth.

Carbon based Guibou · Answer 76 · Thu Mar 15 2018 07:48:32 GMT+0800 (China Standard Time)

@cx405 please open a bugrequest on the nixGL page with some details about your amd driver. I'll contact you directly for tests. (I don't have this kind of hardware, so it will be a kind of ping pong between you and me ;)

cx405 · Answer 77 · Fri Mar 16 2018 01:04:46 GMT+0800 (China Standard Time)

@guibou aye!
/ edit: done :)
// edit: I am currently migrating to nix 18.03 (and have some problems in channel today). I have old 17.09 generation ready to boot into, just in case.
/// edit: I migrated properly to 18.03 and will update the cfg in nixGL issue.

cx405 · Answer 78 · Sat Mar 24 2018 23:44:31 GMT+0800 (China Standard Time)

Wow, I found the reason of my problems with OpenGL in nixos on AMD/Ati cards!
It looks like "radeon" does not suffice in nixos - one needs to specify also "ati" and "vesa": #37673

Philipp Middendorf · Answer 79 · Sat Feb 02 2019 16:18:56 GMT+0800 (China Standard Time)

So still the only working solution is some external tool which needs the current nvidia driver version?

wedens · Answer 80 · Sat Feb 02 2019 16:37:58 GMT+0800 (China Standard Time)

@pmiddend Yeah, https://github.com/guibou/nixGL is still the best solution/workaround we have, unfortunately.

Erin Dahlgren · Answer 81 · Thu May 23 2019 04:31:03 GMT+0800 (China Standard Time)

I had a lot of trouble getting https://github.com/guibou/nixGL to work on my laptop. That's probably because I don't understand my particular cocktail of graphic drivers very well, but then again I don't think most people do. So I spent a couple days coming up with a simpler solution. Because I don't have other hardware to test on, I'd love to know if this works or doesn't work for anyone else.

tl;dr Use libGL indirect rendering through GLX. For why, see note [1] below.

Step 1: Enable indirect glx on an X server not installed by nix (e.g. for Linux, the one installed by default on your host) by running it with the command line option +iglx.

Linux + lightdm (and reboot to restart X): do this.
Linux + anything else (e.g. gdm, kde), unfortunately this is easiest.
macOS + XQuartz: do this.

Step 2: Modern libGL binaries that don't find a graphics driver for direct rendering (e.g. XXX_nvidia.so.0) look for libGLX_indirect.so.0. Make a libGLX_indirect.so.0 symlink that points to libGLX_mesa.so.0 [2]. Then set LD_LIBRARY_PATH to include your libGLX_indirect.so.0 so your libGL dependent binary finds it:

$ nix-shell -p mesa glxinfo

[nix-shell:~]$ find /nix/store -name libGLX_mesa.so.0
/nix/store/bmydwxd7b21315n61fzfzzar6vsafjwn-mesa-noglu-18.3.4-drivers/lib/libGLX_mesa.so.0

[nix-shell:~]$ mkdir libindirect (could be named anything)
[nix-shell:~]$ ln -s /nix/store/bmydwxd7b21315n61fzfzzar6vsafjwn-mesa-noglu-18.3.4-drivers/lib/libGLX_mesa.so.0 libindirect/libGLX_indirect.so.0
[nix-shell:~]$ ls -l libindirect/libGLX_indirect.so.0
libindirect/libGLX_indirect.so.0 -> /nix/store/bmydwxd7b21315n61fzfzzar6vsafjwn-mesa-noglu-18.3.4-drivers/lib/libGLX_mesa.so.0

[nix-shell:~]$ export LD_LIBRARY_PATH=$(pwd)/libindirect:$LD_LIBRARY_PATH

Step 3: Then run an application linked with nix libGL:

[nix-shell:~]$ glxinfo | head
libGL error: unable to load driver: swrast_dri.so
libGL error: failed to load driver: swrast
name of display: :0
display: :0  screen: 0
direct rendering: No (LIBGL_ALWAYS_INDIRECT set)
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_ARB_context_flush_control, GLX_ARB_create_context, 
    GLX_ARB_create_context_profile, GLX_ARB_create_context_robustness, 
    GLX_ARB_fbconfig_float, GLX_ARB_multisample, GLX_EXT_buffer_age, 
    GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile,

Step 4: (optional) To get rid of "libGL error: failed to load driver: swrast" related errors:

[nix-shell:~]$ export LIBGL_ALWAYS_INDIRECT=1

[nix-shell:~]$ glxinfo | head
name of display: :0
display: :0  screen: 0
direct rendering: No (LIBGL_ALWAYS_INDIRECT set)
server glx vendor string: NVIDIA Corporation
server glx version string: 1.4
server glx extensions:
    GLX_ARB_context_flush_control, GLX_ARB_create_context, 
    GLX_ARB_create_context_profile, GLX_ARB_create_context_robustness, 
    GLX_ARB_fbconfig_float, GLX_ARB_multisample, GLX_EXT_buffer_age, 
    GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile,

Notice "direct rendering: No" in the output of glxinfo. That means that you're using indirect gl rendering contexts. If you install glxinfo from your non-nix package manager (e.g. apt, yum) and run it directly on your host, you should see that change to "direct rendering: Yes".

[1] This approach is based on what you'd do if you wanted HW acceleration from a GPU on a remote machine (send rendering requests over the network to the machine with the GPU you want to use). Except in this case you don't go over the network, you just redirect to localhost. If you think about the nix store like a root filesystem on a machine without a GPU --- lacking host-specific graphics drivers, etc --- then I think this makes a lot of sense. Basically it's one level of indirection through an X server to deal with variation in host graphics drivers.

[2] I'll be honest, I'm not a graphics person. So I don't fully understand how generic libGLX_mesa.so.0 is. If it is acting as a generic X client, then I think this solution is host independent. If libGLX_mesa.so.0 is doing something else (and I'm just getting lucky with my setup), then this probably won't work for everyone. Please give it a try and let me know if it works for you --- or provide more info on the limitations of mesa.

[3] Performance: Any sort of indirection has an overhead, so this won't perform the same as direct rendering. There was no noticeable lag with a simple program like glxgears, nor with QtCreator and a simple Qt program. I didn't test games, but if using a local pipe is too slow (I believe that's the default), there's a shared memory extension you can add to most X servers.

[4] Is this stable? Wayland claims to not support indirect contexts, but for backwards compatibility Wayland has XWayland which does. And generally speaking, indirect rendering is such a useful feature that I don't see it disappearing any time soon.

@guibou, @vcunat, @joepie91, @deepfire, @ttuegel --- I'd be interested to know how you feel about this workaround and if it could be made more generic (minus the host X server configuration of course).

Carbon based Guibou · Answer 82 · Thu May 23 2019 09:03:00 GMT+0800 (China Standard Time)

@edahlgren Could you also open an issue on nixGL tracker with details about your configuration?

Erin Dahlgren · Answer 83 · Thu May 23 2019 10:08:12 GMT+0800 (China Standard Time)

@guibou No problem! See this, which also appears to be linked just above.

Schell Carl Scivally · Answer 84 · Fri May 24 2019 01:08:22 GMT+0800 (China Standard Time)

I think I'm running into this issue as well here.

Schell Carl Scivally · Answer 85 · Fri May 24 2019 01:36:30 GMT+0800 (China Standard Time)

Forgive me, but the solution to this problem is a bit hard to parse. I'd like to use nix to build an SDL based game that I can distribute to friends who are running Linux. Will the solutions here work for that case or is this simply the case of running locally for myself?

Erin Dahlgren · Answer 86 · Fri May 24 2019 05:15:50 GMT+0800 (China Standard Time)

@schell You might have a look at nix-bundle (nix -> AppImage), but it's not a complete solution for applications that need hardware rendering through a system libGL (links to this issue). You could use software rendering for libGL, but that might not be fast enough, I don't know. Otherwise the solutions above I believe are for nix users.

@guibou @vcunat IIUC, nixGL will force applications to depend on specific graphics drivers and libGL libraries (that happen to match system requirements), which probably depend on their own version of libc. The application might be packaged with a very specific version of libc. Which one wins? The libc needed by the graphics driver? If so, is that safe (application developers aren't expecting a change)? Or perhaps it is unwise for graphics applications to pin to a specific version of libc in the first place (nix seems to promote pinning), given expected libGL system variation for desktop apps?

Schell Carl Scivally · Answer 87 · Sat May 25 2019 00:58:51 GMT+0800 (China Standard Time)

Thanks @edahlgren - for now I'll continue building with stack for desktop and use nix to wrangle ghcjs.

Vladimír Čunát · Answer 88 · Sun May 26 2019 15:43:12 GMT+0800 (China Standard Time)

Pulling other libs via the C linker does cause issues sometimes, so it really helps to minimize the set; glibc libs actually provide very high level of compatibility (at least in the direction of increasing the version between build-time and run-time), but some libGL pull even libstdc++ or libwayland IIRC and these do cause problems occasionally if the versions differ significantly (they tend to have good ABI versioning, so the error should be loud and clear at least).

I'm afraid my GL knowledge is very low; I don't really have an idea about difference between (in)direct variants. I started my nixpkgs GL involvement in a time when there were only 5-10 active contributors in whole nixpkgs.

Carbon based Guibou · Answer 89 · Tue May 28 2019 01:42:30 GMT+0800 (China Standard Time)

@edahlgren The idea with nixGL is to install a nixGL which correctly fits with your system. All the userspace driver will be provided by nix.

Unfortunately, for now, nixGL does not do any detection of the runtime setup and you need to setup it by yourself.

Tyson Whitehead · Answer 90 · Tue May 28 2019 05:25:20 GMT+0800 (China Standard Time)

@edahlgren just wondering about the indirect thing as have recently spent a lot of time on OpenGL issues for our system (Compute Canada). Are you using X11 forwarding? I ask as I understand it tries to load libglx_$vendor where $vendor is the value reported in the GLX_EXT_libglvnd extension.

In our testing we noticed that enabling the indirect rendering (client sending the GL calls to the X11 sever for rendering) was something required by users using the Mac X11 server (and I expect Windows MobaXterm user as well) because it didn't support the X11 visuals required for the mesa client to perform client side software rendering (simulate GL by rendering them in software to a 2D X11 visual).

This was evident by the client program (glxinfo, glxgears, etc.) printing out a message about unsupported visuals when running over X11 forwarding from a non-linux machine. All the linux hosts we connected from did not have this issue and did not report this message. I'm presuming recent X11 servers offer the visuals required for mesa to do it client side software rendering.

I mention this as I understand indirect rendering can involve a lot of round trips between the client and the X11 server depending on the application, which can really kill performance over higher latency connections.

The best solution we found for situations where you just need to handle X11 connections was to use a mesa compiled with glx=gallium-xlib. This gives pure client-side software rendering. It is not without its own pain though as it severely restricts what other options you can turn on when building mesa (e.g., no EGL support, no libvglvnd support, etc.).

Still doing some work on this (currently sidetracked no other duties), but here is the basic derivation so far

mesa_glxgallium = stdenv.mkDerivation rec {
  version = super.mesa_noglu.version;
  name = "mesa-glxgallium-${version}";
  src = super.mesa_noglu.src;

  outputs = [ "out" "dev" ];

  configureFlags = [
    "--with-gallium-drivers=swrast"
    "--with-platforms=x11"           # surfaceless would make sense, but egl requires dri
    "--disable-dri"
    "--enable-glx=gallium-xlib"      # gallium-xlib requires no dri
    "--enable-gallium-osmesa"
    "--enable-llvm"
    "--disable-egl"                  # egl requries dri for some reason
    "--disable-gbm"                  # gbm requires dri for some reason
    "--enable-llvm-shared-libs"
    "--disable-opencl"
  ];
  nativeBuildInputs = [
    autoreconfHook pkgconfig
    python2
  ];
  propagatedBuildInputs = [ ];
  buildInputs = [
    llvmPackages.llvm
    xorg.xorgproto xorg.libX11 xorg.libXext xorg.libxcb
    expat
  ];

  enableParallelBuilding = true;

  meta = with stdenv.lib; {
    description = "An open source implementation of OpenGL";
    homepage = https://www.mesa3d.org/;
    license = licenses.mit; # X11 variant, in most files
    platforms = platforms.linux;
  };
};

Erin Dahlgren · Answer 91 · Tue May 28 2019 12:42:28 GMT+0800 (China Standard Time)

@twhitehead If you mean X11 forwarding in the sense of ssh -X, no. I'm just enabling +iglx on a local X server. In my setup, local X clients link to libGLX_indirect.so which happens to symlink to libGLX_mesa.so. These local X clients use libGLX_mesa.so to send GL requests to the X server using IPC (kernel pipe or shared memory). The X server links to host GL libs (e.g. nvidia). Doing things this way is a bit absurd given that it's all local, but the indirection acts as a boundary between nix applications and the system. In this setup, there isn't a high latency connection because there's no remote machine.

Regarding needing indirect rendering for pure software rendering to work on macOS --- that's good to know! I sent this email 2 days ago to the mesa users mailing list. It asks about building gallium on/for macOS and whether there's support for gallium on macOS without X, like how they support GDI for Windows. My goal here is to offer software rendering as a basic working option for nix users if configuring an X server or graphics drivers is too cumbersome. I'm trying to get hold of cheap, used macOS laptops or servers to test.

As for libglvnd, I ran into that too ☹️ On Linux at least, libglvnd seems to ask the X server which libGXL_vendor.so to load, like you say. So if you're like me and have nvidia drivers installed on your system, libglvnd will blindly try to use them, even if you want to use pure software rendering instead. To get around this, I built a gallium libGL.so from scratch and linked simple nix GUIs (e.g. qt apps, editors) to that. Here's my derivation, which looks similar to yours:

{ stdenv, fetchurl, pkgconfig
, scons, flex_2_5_35, bison2
, python27, python27Packages
, llvm_35, xorg, expat
}:

stdenv.mkDerivation rec {
  name    = "swr-${version}";
  version = "19.0.5";

  src = fetchurl {
    url    = "https://mesa.freedesktop.org/archive/mesa-${version}.tar.gz";
    sha256 = "0w2ff9gzahg4djmqmb903gm9bqqcs1x8piw7g0g5vhdy4f6bgrmn";
  };

  nativeBuildInputs = [
    pkgconfig scons flex_2_5_35 bison2
    python27 python27Packages.Mako
  ];

  buildInputs = [
    llvm_35 xorg.libX11 xorg.libXext
    xorg.libXdamage xorg.libXfixes expat
  ];

  phases = [ "unpackPhase" "buildPhase" "installPhase" ];
  
  buildPhase = ''
    scons build=release libgl-xlib
  '';

  installPhase = ''
    mkdir -p $out/lib
    cp build/linux-x86_64/gallium/targets/libgl-xlib/* $out/lib/
  '';
}

You use it like this (substitute qtcreator and qt5Full packages with your GUI app):

$ nix-shell -p swr qtcreator qt5Full
[nix-shell:~]$ export LD_LIBRARY_PATH=/nix/store/xhzgpc7rgj760vfxk5qad6iqaaw519qg-swr-19.0.5/lib
[nix-shell:~]$ qtcreator

I would have taken your approach (super.mesa_noglu) but I wanted to try building the recommended way with scons. See the gallium project page for more details. The derivation could probably be much improved by people with more experience than me.

Regarding this severely restricting other options you can build with (e.g. EGL, libvglvnd) maybe that's OK. One can have multiple libGL.so libraries in one's nix store for different needs. Some package could try to bridge them all together into the "one true libGL.so", but I find it important to have the option to just use one separately in a simple way (like software rendering only). As for libglvnd, my understanding after reading this thread is that it's more for handling optimus setups (e.g. intel + nvidia) than being a generic dispatching library (see comment 8). Perhaps someone can correct or validate that.

@guibou Cool, right. I guess I'm curious, is there still a chance that the drivers nixGL installs can somehow link poorly with applications pinned to specific versions of libstdc++, etc? I guess that'd be true if the drivers can't be compiled from source using the same standard libs and compiler as the application?

@vcunat Got it, thanks 😄 My goal here is to make it very easy for the academics I work with to use and build nix GUI apps so they can make their research results (which sometimes depend on GUIs) more reproducible and friendly to people creating derivative work. They rarely have the patience or knowledge to debug their development system like we're doing in this thread. They might eventually become NixOS users, but not at the beginning.

I've been thinking more about the problem in this thread in general, curious to hear more thoughts. It seems that nix, AppImage, docker, chroot, and all other things that try to statically define or contain applications suffer from the same problem. That there are systems libraries you link to (e.g. libGL.so) that you can't statically link to ahead of time because they depend on variable hardware.

Like others have said, in many cases the ABIs of these libraries and their dependencies are stable. But the set of possible libGL.so dependencies (e.g. libc, libz, libxcb, libstdc++) isn't well specified or stable as far as I know. So application developers seem to build with old versions of their compiler and pray. That seems somewhat contrary to nix: with nix, you're encouraged to be very specific about versions of libraries so your app or experiment is reproducible for others (a great thing). And in practice, libGL.so (and other systems libs) can make that impossible or fragile at best.

Instead, I think what I want is libGL.so and all other systems libraries I don't know about (but need in the same way I need the Linux kernel) to be built into a separate process that uses IPC to communicate with dependent apps (shared memory for IPC, ideally). That means there's no linking conflicts between system libs and apps. And if the ABI of libGL is stable, in this model that's the only ABI I need to care about. Indirect rendering through an X server is one way to do this. But it could probably be a lot simpler than that. The only problem is that I don't know how to achieve it yet.

Carbon based Guibou · Answer 92 · Tue May 28 2019 21:22:33 GMT+0800 (China Standard Time)

@edahlgren

Cool, right. I guess I'm curious, is there still a chance that the drivers nixGL installs can somehow link poorly with applications pinned to specific versions of libstdc++, etc? I guess that'd be true if the drivers can't be compiled from source using the same standard libs and compiler as the application?

I'm not sure I understand your question. The driver that nixGL installs link properly with all the needed dependencies (including libc, libstdc++, ...) taken directly from nix. That's the point of nixGL. In this case, it will have the same libraries as the application.

Erin Dahlgren · Answer 93 · Wed May 29 2019 03:21:04 GMT+0800 (China Standard Time)

@guibou Sorry if I wasn't clear. What I mean is that nixGL seems to download nvidia drivers if needed, and those are closed source IIUC. Those presumably have certain fixed dependencies. They might use versions of libc, libstdc++, etc taken from nix. But that doesn't prevent versioning conflicts with nix apps that depend on other versions of libc, libstdc++, etc taken from nix too, correct?

But maybe that's just a general problem of linking to closed source libraries, not specific to nix (though maybe more obvious with nix).

Timothée Jourde · Answer 94 · Sat Jun 01 2019 01:41:11 GMT+0800 (China Standard Time)

Hi there! I packaged an OpenGL application with Nix. It's working well on NixOS, but now I would like to distribute it as an AppImage with nix-bundle. I successfully generated the image and I can run it on Ubuntu 18, but it rapidly fails with a GLX error from GLFW (same thing if I install Nix and import the closure). I tried all of the above hacks without any success. Is there any chance I can get it running outside of NixOS without root access (so I can easily distribute it)? Maybe I could bundle nixGL?

Carbon based Guibou · Answer 95 · Sat Jun 01 2019 03:18:06 GMT+0800 (China Standard Time)

@timjrd you won't be able to bundle nixGL because for now, nixGL must know the targeted driver at buildtime.

They might use versions of libc, libstdc++, etc taken from nix. But that doesn't prevent versioning conflicts with nix apps that depend on other versions of libc, libstdc++, etc taken from nix too, correct?

That's why you MUST use nixgl with the same nixpkgs clone as the one used for your application. The nixGL derivation does have a pkgs attribute which MUST be set to the same nixpkgs clone as the one used by your application.

Timothée Jourde · Answer 96 · Tue Jun 04 2019 23:58:40 GMT+0800 (China Standard Time)

OK, I eventually managed to get my AppImage "automatically" working on Ubuntu 18 using patchelf, LD_LIBRARY_PATH, and LIBGL_DRIVERS_PATH in a hacky wrapper script. It should be extendable to other distributions, although it's not very robust. This helper script was very useful to find missing dlopened libraries.

Niklas Hambüchen · Answer 97 · Fri Oct 11 2019 20:06:11 GMT+0800 (China Standard Time)

For meshlab, nixGL continues to be a workaround: #70937 (comment)

Adieu · Answer 98 · Fri May 01 2020 22:48:21 GMT+0800 (China Standard Time)

Hit this issue in crostini (linux environment in ChromeOS).

Fixed it with

sudo mkdir -p /run/opengl-driver/;sudo ln -s `nix eval --raw nixpkgs.mesa_drivers.outPath`/lib /run/opengl-driver/lib

Might be useful for other environments too.

Carbon based Guibou · Answer 99 · Sat May 02 2020 00:03:51 GMT+0800 (China Standard Time)

@adieu nixGL automate this process and does not need sudo.

nixos-discourse · Answer 100 · Sun May 10 2020 06:26:29 GMT+0800 (China Standard Time)

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/nixgl-needs-you-for-testing/6921/7