Curl flake with ^out,dev not adding dynamic libraries

Question

Curl flake with ^out,dev not adding dynamic libraries

undecided opened this issue 3 months ago · comments

Matthew Bennett-Lovesey commented 3 months ago

What happened?

Having just worked around #1851 and deleted my /nix and reinstalled everything, suddenly my project is broken thanks to ruby's FFI being unable to find libcurl.

Looks like even with the curl line added, .devbox/nix/profile/default/lib (as per $LD_LIBRARY_PATH) does not contain libcurl

Steps to reproduce

With export DEVBOX_USE_VERSION=0.8.7 run the script devbox run problem - it works
unset DEVBOX_USE_VERSION and with the latest devbox, run the same script - it breaks
With 0.9.2-dev it also breaks (with new-style glibc-patch and --output out,dev, of course!)

Command

run, shell

devbox.json

{
  "packages": [
    "ruby@latest",
    "github:nixos/nixpkgs#curl^out,dev"
  ],
  "shell": {
    "scripts": {
      "problem": "gem install typhoeus ; ruby -rtyphoeus -e 'puts :hi'"
    }
  }
}

Devbox version

0.9.1

Nix version

nix (Nix) 2.18.1

What system does this bug occur on?

Linux (x86-64)

Debug logs

❯ DEVBOX_USE_VERSION=0.8.7 devbox run problem
Info: New devbox available: 0.8.7 -> v0.9.1. Please run `devbox version update`.
Ensuring packages are installed.

Installing package: curl.

[1/1] curl
[1/1] curl: Success
✓ Computed the Devbox environment.
Successfully installed typhoeus-1.4.1
Parsing documentation for typhoeus-1.4.1
Done installing documentation for typhoeus after 0 seconds
1 gem installed
hi

❯ DEVBOX_USE_VERSION=0.9.1 devbox run problem
Ensuring packages are installed.
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
Successfully installed typhoeus-1.4.1
Parsing documentation for typhoeus-1.4.1
Done installing documentation for typhoeus after 0 seconds
1 gem installed
/home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library': Could not open library 'libcurl': libcurl: cannot open shared object file: No such file or directory. (LoadError)
Could not open library 'libcurl.so': libcurl.so: cannot open shared object file: No such file or directory.
Could not open library 'libcurl.so.4': libcurl.so.4: cannot open shared object file: No such file or directory.
Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:95:in `block in ffi_lib'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `map'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ffi-1.16.3/lib/ffi/library.rb:94:in `ffi_lib'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:10:in `<module:Curl>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:3:in `<module:Ethon>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curls/settings.rb:2:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:28:in `<module:Curl>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:14:in `<module:Ethon>'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon/curl.rb:9:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/ethon-0.16.0/lib/ethon.rb:16:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
	from /home/matthew/projects/third_party/devboxplay/.devbox/virtenv/ruby/gems/typhoeus-1.4.1/lib/typhoeus.rb:2:in `<top (required)>'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:141:in `rescue in require'
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:135:in `require'
<internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require': cannot load such file -- typhoeus (LoadError)
	from <internal:/nix/store/dmflx3hwdk6iyv8c1k5fz8vdh8d6zgxv-ruby-3.3.0/lib/ruby/3.3.0/rubygems/core_ext/kernel_require.rb>:136:in `require'
Error: error running script "problem" in Devbox: exit status 1

Also, this might be useful:

❯ DEVBOX_USE_VERSION=0.8.7 devbox run 'ls $LD_LIBRARY_PATH'
Info: New devbox available: 0.8.7 -> v0.9.1. Please run `devbox version update`.
libcurl.la  libcurl.so	libcurl.so.4  libcurl.so.4.8.0	libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby
 
❯ DEVBOX_USE_VERSION=0.9.1 devbox run 'ls $LD_LIBRARY_PATH'
Ensuring packages are installed.
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Changing the devbox.json to force it to reinstall (e.g. changing github:nixos/ to github:NixOS/) gives this:

❯ DEVBOX_USE_VERSION=0.9.1 devbox run 'ls $LD_LIBRARY_PATH'
Ensuring packages are installed.
[1/1] curl
[1/1] curl: Success
✓ Computed the Devbox environment.
Info: Removing curl@8.6.0
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Greg Curtis · Answer 1 · Wed Feb 28 2024 00:47:47 GMT+0800 (China Standard Time)

Thanks for the thorough bug report. I'll take a look at this today.

Greg Curtis · Answer 2 · Wed Feb 28 2024 06:09:33 GMT+0800 (China Standard Time)

@savil it looks like we're dropping the out output from curl when it's specified as a flake reference. I think this might be a regression after some of the recent lock file changes.

Here's what I see happening with the Devbox nix profile using the devbox.json from the issue (logs snipped to only include relevant messages).

Enter a shell, see that only ruby and curl^dev are installed to the profile (curl^out is missing).

$ devbox version -v
Version:     0.0.0-dev
Platform:    linux_arm64
Commit:      0662f451222fe3195127f6376d49a8a7239192d3
Go Version:  go1.22.0
$ devbox shell
2024/02/27 16:49:15 running command: /run/current-system/sw/bin/nix profile install --profile /home/gcurtis/devbox/nixosvm/issue1852/.devbox/nix/profile/default --impure --priority 6 --extra-experimental-features ca-derivations --option experimental-features nix-command flakes fetch-closure --offline /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0
2024/02/27 16:49:15 running command: /run/current-system/sw/bin/nix profile install --profile /home/gcurtis/devbox/nixosvm/issue1852/.devbox/nix/profile/default --impure --priority 7 --extra-experimental-features ca-derivations --option experimental-features nix-command flakes fetch-closure --offline /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

See that the curl libraries are missing from the profile's lib directory.

(devbox) $ ls .devbox/nix/profile/default/lib
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

See that only curl^dev is installed to the profile.

(devbox) $ nix profile list --profile .devbox/nix/profile/default
Index:              0
Store paths:        /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0

Index:              1
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

Manually add nixpkgs#curl^out,dev to the profile and see that libcurl appears in the profile's lib directory.

(devbox) $ nix profile install --profile .devbox/nix/profile/default nixpkgs#curl^out,dev
(devbox) $ nix profile list --profile .devbox/nix/profile/default
Index:              0
Store paths:        /nix/store/0dqds3mklwg3jygw6gwi7dknag2q7fik-ruby-3.3.0

Index:              1
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev

Index:              2
Flake attribute:    legacyPackages.aarch64-linux.curl^dev,out
Original flake URL: flake:nixpkgs
Locked flake URL:   github:NixOS/nixpkgs/8520c158aee718c6e87b56881105fc4223c3c723
Store paths:        /nix/store/47wjvvb6smgy33p75xnr6liif4xrrl9z-curl-8.6.0-dev /nix/store/ilzq62jr6sj7p2vlq0sjsmg1264w1vgf-curl-8.6.0
(devbox) $ ls .devbox/nix/profile/default/lib
libcurl.la  libcurl.so	libcurl.so.4  libcurl.so.4.8.0	libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Exit and re-enter the devbox shell. See that devbox removed curl^out (but not curl^dev) from the profile.

(devbox) $ exit
$ devbox shell
2024/02/27 17:01:24 Removing packages from nix profile: curl@8.6.0

See that libcurl is missing again.

(devbox) $ ls .devbox/nix/profile/default/lib
libruby-3.3.0.so  libruby-3.3.0.so.3.3	libruby-3.3.0.so.3.3.0	pkgconfig  ruby

Matthew Bennett-Lovesey · Answer 3 · Wed Feb 28 2024 18:15:15 GMT+0800 (China Standard Time)

Thanks for investigating. Off the back of that, in case it's any use to anyone else, I've been able to add the following to my devbox.json as a workaround, and it seems to work nicely

  "shell": {
    "init_hook": [
      "nix profile install --profile .devbox/nix/profile/default nixpkgs#curl^out,dev --extra-experimental-features nix-command --extra-experimental-features flakes"

With that, when I do devbox shell, it's all installed correctly

❯ devbox version
0.9.1

❯ ls .devbox/nix/profile/default/lib/*curl*
.devbox/nix/profile/default/lib/libcurl.la  .devbox/nix/profile/default/lib/libcurl.so.4
.devbox/nix/profile/default/lib/libcurl.so  .devbox/nix/profile/default/lib/libcurl.so.4.8.0