Nettle.jl 0.5.0, Julia 1.6, and Windows

Question

Nettle.jl 0.5.0, Julia 1.6, and Windows

BioTurboNick opened this issue 3 years ago · comments

One of my packages has been getting failures on Travis-CI specifically with Julia 1.6. Now that 1.6 is in beta, starting to investigate.

It works fine on my local Windows computer, so it might just be Travis.

The error I'm getting:

ERROR: LoadError: InitError: could not load library "C:\Users\travis\.julia\artifacts\5fe2f6efc34d15bcfdb7300a589d66a933a5c946\bin\libhogweed-4.dll"
Access is denied. 
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl .\libdl.jl:114
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl .\libdl.jl:114
  [3] macro expansion
    @ ~\.julia\packages\JLLWrappers\WnWcZ\src\products\library_generators.jl:63 [inlined]
  [4] __init__()
    @ Nettle_jll ~\.julia\packages\Nettle_jll\Z5Qas\src\wrappers\x86_64-w64-mingw32.jl:11
[...trimmed for length...]
during initialization of module Nettle_jll
in expression starting at C:\Users\travis\.julia\packages\Nettle\LhcpB\src\Nettle.jl:1

Jesse Campbell · Answer 1 · Sat Jan 23 2021 12:12:15 GMT+0800 (China Standard Time)

I'm seeing the same issue when trying to precompile Genie v1.12.0 which imports Nettle v0.5.0.

It looks like it can't load the libhodgweed-4.dll library. I'm working on my local Windows 10 PC, also running Julia 1.6.0-beta1, so I think this isn't just a Travis-CI issue.

(website) pkg> st
Project website v0.1.0
Status `<project_path>\Project.toml`
  [c43c736e] Genie v1.12.0
  [49dea1ee] Nettle v0.5.0

(website) pkg> precompile
Precompiling project...
  Progress [========================================>]  2/2
  ✗ Nettle
  ✗ Genie
0 dependencies successfully precompiled in 9 seconds (32 already precompiled)

ERROR: The following 2 direct dependencies failed to precompile:

Nettle [49dea1ee-f6fa-5aa6-9a11-8816cee7d4b9]

ERROR: LoadError: InitError: could not load library "~\.julia\artifacts\5fe2f6efc34d15bcfdb7300a589d66a933a5c946\bin\libhogweed-4.dll"
Access is denied.
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl .\libdl.jl:114
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl .\libdl.jl:114
  [3] macro expansion
    @ ~\.julia\packages\JLLWrappers\WnWcZ\src\products\library_generators.jl:63 [inlined]
  [4] __init__()
    @ Nettle_jll ~\.julia\packages\Nettle_jll\Z5Qas\src\wrappers\x86_64-w64-mingw32.jl:11
  [5] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base .\loading.jl:670
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base .\loading.jl:756
  [7] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:994
  [8] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:910
  [9] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:897
 [10] include
    @ .\Base.jl:386 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1209
 [12] top-level scope
    @ none:1
 [13] eval
    @ .\boot.jl:360 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude .\client.jl:446
 [15] top-level scope
    @ none:1
during initialization of module Nettle_jll
in expression starting at ~\.julia\packages\Nettle\LhcpB\src\Nettle.jl:1

Genie [c43c736e-a2d1-11e8-161f-af95117fbd1e]

ERROR: LoadError: InitError: could not load library "~\.julia\artifacts\5fe2f6efc34d15bcfdb7300a589d66a933a5c946\bin\libhogweed-4.dll"
Access is denied.
Stacktrace:
  [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
    @ Base.Libc.Libdl .\libdl.jl:114
  [2] dlopen(s::String, flags::UInt32)
    @ Base.Libc.Libdl .\libdl.jl:114
  [3] macro expansion
    @ ~\.julia\packages\JLLWrappers\WnWcZ\src\products\library_generators.jl:63 [inlined]
  [4] __init__()
    @ Nettle_jll ~\.julia\packages\Nettle_jll\Z5Qas\src\wrappers\x86_64-w64-mingw32.jl:11
  [5] _include_from_serialized(path::String, depmods::Vector{Any})
    @ Base .\loading.jl:670
  [6] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String)
    @ Base .\loading.jl:756
  [7] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:994
  [8] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:910
  [9] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:897
 [10] include
    @ .\Base.jl:386 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::String)
    @ Base .\loading.jl:1209
 [12] top-level scope
    @ none:1
 [13] eval
    @ .\boot.jl:360 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude .\client.jl:446
 [15] top-level scope
    @ none:1
during initialization of module Nettle_jll
in expression starting at ~\.julia\packages\Nettle\LhcpB\src\Nettle.jl:1
ERROR: LoadError: LoadError: Failed to precompile Nettle [49dea1ee-f6fa-5aa6-9a11-8816cee7d4b9] to ~\.julia\compiled\v1.6\Nettle\jl_6034.tmp.
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:33
  [2] compilecache(pkg::Base.PkgId, path::String, internal_stderr::IOContext{Base.PipeEndpoint}, internal_stdout::IOContext{IOStream})
    @ Base .\loading.jl:1356
  [3] compilecache(pkg::Base.PkgId, path::String)
    @ Base .\loading.jl:1302
  [4] _require(pkg::Base.PkgId)
    @ Base .\loading.jl:1017
  [5] require(uuidkey::Base.PkgId)
    @ Base .\loading.jl:910
  [6] require(into::Module, mod::Symbol)
    @ Base .\loading.jl:897
  [7] include(mod::Module, _path::String)
    @ Base .\Base.jl:386
  [8] include(x::String)
    @ Genie ~\.julia\packages\Genie\fkVHD\src\Genie.jl:4
  [9] top-level scope
    @ ~\.julia\packages\Genie\fkVHD\src\Genie.jl:31
 [10] include
    @ .\Base.jl:386 [inlined]
 [11] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
    @ Base .\loading.jl:1209
 [12] top-level scope
    @ none:1
 [13] eval
    @ .\boot.jl:360 [inlined]
 [14] eval(x::Expr)
    @ Base.MainInclude .\client.jl:446
 [15] top-level scope
    @ none:1
in expression starting at ~\.julia\packages\Genie\fkVHD\src\Encryption.jl:1
in expression starting at ~\.julia\packages\Genie\fkVHD\src\Genie.jl:1


(website) pkg>

System info for context

julia> versioninfo()
Julia Version 1.6.0-beta1.0
Commit b84990e1ac (2021-01-08 12:42 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-11.0.0 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = "C:\Program Files\Microsoft VS Code\Code.exe"
  JULIA_NUM_THREADS =

Jesse Campbell · Answer 2 · Sat Jan 23 2021 12:56:14 GMT+0800 (China Standard Time)

Update: looks like I just needed to adjust my terminal's permissions lol - as soon as I "ran as Administrator" everything ran just fine.

Could be related to this issue? JuliaPackaging/Yggdrasil#1969

Nicholas Bauer · Answer 3 · Fri Mar 19 2021 04:53:11 GMT+0800 (China Standard Time)

@staticfloat Verdict if it's a Nettle issue or Julia 1.6 issue or Travis issue? Then I'll know who to bother. :-)

Elliot Saba · Answer 4 · Fri Mar 19 2021 04:58:06 GMT+0800 (China Standard Time)

Can you verify that it happens on 1.6-rc2 from a fresh depot? E.g. run JULIA_DEPOT_PATH=<some temp path> julia then install Nettle and ensure it fails in this manner?

Nicholas Bauer · Answer 5 · Fri Mar 19 2021 05:45:20 GMT+0800 (China Standard Time)

Sorry for the premature post. Thought I had confirmed it was only on Travis but nope, my local system had a problem too.

It is 1.6.0-rc3

Trying your suggestion

Nicholas Bauer · Answer 6 · Fri Mar 19 2021 05:47:19 GMT+0800 (China Standard Time)

Yes, Nettle fails to install when I use a temporary directory.

Elliot Saba · Answer 7 · Sat Mar 20 2021 10:56:03 GMT+0800 (China Standard Time)

Okay, I found the issue, and it is quite simple to fix. We just need to regenerate a new version of Nettle over at Yggdrasil. I have a PR here, but for some reason it failed on a few platforms; just need to shepherd that through.

Elliot Saba · Answer 8 · Sun Mar 21 2021 01:15:28 GMT+0800 (China Standard Time)

Can you ]update and try again?

Nicholas Bauer · Answer 9 · Sun Mar 21 2021 03:14:03 GMT+0800 (China Standard Time)

Hmm, it installs okay, but I'm getting an error inside Nettle now, affecting all versions of Julia and platforms:

ERROR: could not load symbol "nettle_hashes":
The specified procedure could not be found.
Stacktrace:
 [1] get_hash_types()
   @ Nettle ~\.julia\packages\Nettle\LhcpB\src\hash_common.jl:51
 [2] HMACState(name::String, key::Vector{UInt8})
   @ Nettle ~\.julia\packages\Nettle\LhcpB\src\hmac.jl:17
 [3] digest(hmac_name::String, key::Vector{UInt8}, data::Vector{UInt8})
   @ Nettle ~\.julia\packages\Nettle\LhcpB\src\hmac.jl:50

Nicholas Bauer · Answer 10 · Sun Mar 21 2021 03:36:13 GMT+0800 (China Standard Time)

I'm not super familiar with libnettle, but I think they added an underscore at some point? (from a quick look around) https://git.lysator.liu.se/nettle/nettle/-/commit/8bf4747d9cca7fd2cdd6ca8e14bce7cb334b2a7a

unsafe_load(cglobal(("_nettle_hashes",libnettle),Ptr{Ptr{Cvoid}}),hash_idx)

There's a macro they suggest using instead that doesn't need the underscore, but is it possible to use from Julia? https://git.lysator.liu.se/nettle/nettle/-/blob/master/nettle.texinfo

Nicholas Bauer · Answer 11 · Mon Mar 22 2021 08:16:58 GMT+0800 (China Standard Time)

Confirmed, thanks!