tpapp / DynamicHMC.jl

Implementation of robust dynamic Hamiltonian Monte Carlo methods (NUTS) in Julia.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Pkg.test("DynamicHMC") on Julia 1.5.0 gives two fails

MooersLab opened this issue · comments

Running on a MacOS 10.15.
I was able to work the worked example in the documentation,
so I am unsure if these test failures are important.

Test Summary:     | Pass  Total
simulated meancov |    2      2
Test Summary: |    Pass    Total
test-trees.jl | 2003798  2003798
leapfrog back and forth: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
  Expression: (z.p, z1.p, norm = (x->begin
                norm(x, Inf)
            end), rtol = 0.001)
   Evaluated: [-0.7522839018993613, 0.2728057174736127]  [-0.7508500824628771, 0.271725160229739] (norm=#20, rtol=0.001)
Stacktrace:
 [1] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
 [3] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:144
leapfrog back and forth: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
  Expression: (z.p, z1.p, norm = (x->begin
                norm(x, Inf)
            end), rtol = 0.001)
   Evaluated: [-0.02532229751461036, 0.030292707711202047]  [-0.027146248766058756, 0.03115685279645941] (norm=#20, rtol=0.001)
Stacktrace:
 [1] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:171
 [2] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
 [3] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/test-hamiltonian.jl:144
Test Summary:                                    |  Pass  Fail  Total
test-hamiltonian.jl                              | 13260     2  13262
  Gaussian KE full                               |   400          400
  Gaussian KE diagonal                           |   400          400
  phasepoint internal consistency                |    23           23
  leapfrog calculation                           |   205          205
  leapfrog Hamiltonian invariance                | 10000        10000
  leapfrog back and forth                        |  2198     2   2200
  PhasePoint building blocks and infinite values |     9            9
  Hamiltonian and KE printing                    |     3            3
  unit normal simple HMC                         |     2            2
  normal NUTS HMC transition mean and cov        |    20           20
ERROR: LoadError: Some tests did not pass: 13260 passed, 2 failed, 0 errored, 0 broken.
in expression starting at /Users/blaine/.julia/packages/DynamicHMC/83Q9C/test/runtests.jl:17
ERROR: Package DynamicHMC errored during testing

Thanks for reporting. This probably comes from a change in Julia's RNG producing different random values for the tests, which happen to be a bit more ill-conditioned than previously. I will investigate and fix, but this should be innocuous.

Fixed on latest master.

Testing of todays version on github in Julia 15.1 on Mac OS X 15.6 progresses further, but fails on the testing of sample-correctness-utitlies.jl.

Test Summary: | Pass Total
simulated meancov | 2 2
Test Summary: | Pass Total
test-trees.jl | 2003798 2003798
Test Summary: | Pass Total
test-hamiltonian.jl | 13262 13262
Test Summary: | Pass Total
test-NUTS.jl | 32 32
Test Summary: | Pass Total
test-stepsize.jl | 1132 1132
Test Summary: | Pass Total
test-mcmc.jl | 13 13
Test Summary: | Pass Total
test-diagnostics.jl | 20 20
INFO while testing: mixture of two normals, dimension 3
ALERT some τ = [0.169, 0.184, 0.147] < 0.15
INFO while testing: elongate(1.2, 𝑁), dimension 5
ALERT extreme Anderson-Darling p ≈ 1.66e-14 < 0.0 for coordinate 1 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 1.6572882297252903e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 7.98e-14 < 0.0 for coordinate 2 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 7.975948103200937e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 2.39e-14 < 0.0 for coordinate 3 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 2.386491241178068e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 1.56e-16 < 0.0 for coordinate 4 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 1.555009605115904e-16 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 5.94e-13 < 0.0 for coordinate 5 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 5.938619232986046e-13 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/sample-correctness-tests.jl:75
Test Summary: | Pass Fail Total
sample-correctness-tests.jl | 153 5 158
NUTS tests with random normal | 96 96
NUTS tests with specific normal distributions | 40 40
NUTS tests with mixtures | 6 6
NUTS tests with heavier tails and skewness | 11 5 16
ERROR: LoadError: Some tests did not pass: 153 passed, 5 failed, 0 errored, 0 broken.
in expression starting at /Users/blaine/.julia/packages/DynamicHMC/fy0Ju/test/runtests.jl:27
ERROR: Package DynamicHMC errored during testing

Thanks. Is this the master branch of DynamicHMC?

I am happy to address this but I can't replicate on Linux and Julia 1.5 or 1.6 --- I don't have access to OS X.

PRs are appreciated, just extend the bounds that fail by a minor amount.

Is this still an open issue?

I installed julia 1.5.3 on Ubuntu and DyamicHMC and ran test on it. All was well.

On the Mac 10.15.7 with julia 1.5.3 and DymaicHMC from the default distribution installed, test returns

┌ Info: MCMC progress
│ step = 9801
│ seconds_per_step = 5.0e-6
└ estimated_seconds_left = 0.00099
┌ Info: MCMC progress
│ step = 9901
│ seconds_per_step = 5.4e-6
└ estimated_seconds_left = 0.00054
[ Info:
┌ Info: Starting MCMC
└ total_steps = 1000
[ Info:
┌ Info: MCMC progress
│ step = 1
│ seconds_per_step = 0.018
└ estimated_seconds_left = 18.0
┌ Info: Starting MCMC
└ total_steps = 1000
[ Info:
[ Info:
┌ Info: MCMC progress
│ step = 1
│ seconds_per_step = 4.0e-5
└ estimated_seconds_left = 0.04
Test Summary: |
test-logging.jl | No tests
┌ Warning: (t::AbstractTransform)(x) is deprecated, use transform(t, x) instead
│ caller = rand_C(::Int64) at common.jl:118
└ @ Main ~/.julia/packages/DynamicHMC/n1LsS/test/common.jl:118
INFO while testing: mixture of two normals, dimension 3
ALERT some τ = [0.169, 0.184, 0.147] < 0.15
INFO while testing: elongate(1.2, 𝑁), dimension 5
ALERT extreme Anderson-Darling p ≈ 1.66e-14 < 0.0 for coordinate 1 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 1.6572882297252903e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 7.98e-14 < 0.0 for coordinate 2 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 7.975948103200937e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 2.39e-14 < 0.0 for coordinate 3 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 2.386491241178068e-14 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 1.56e-16 < 0.0 for coordinate 4 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 1.555009605115904e-16 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75
ALERT extreme Anderson-Darling p ≈ 5.94e-13 < 0.0 for coordinate 5 of 5
NUTS tests with heavier tails and skewness: Test Failed at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
Expression: p_AD ≥ pd_fail
Evaluated: 5.938619232986046e-13 ≥ 2.0000000000000004e-7
Stacktrace:
[1] NUTS_tests(::Random._GLOBAL_RNG, ::LogDensityTestSuite.Elongate{StandardMultivariateNormal,Float64}, ::String, ::Int64; K::Int64, io::Base.TTY, mcmc_args::NamedTuple{(),Tuple{}}, R̂_alert::Float64, R̂_fail::Float64, τ_alert::Float64, τ_fail::Float64, p_alert::Float64, p_fail::Float64, EBFMI_alert::Float64, EBFMI_fail::Float64) at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-utilities.jl:115
[2] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:78
[3] top-level scope at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115
[4] top-level scope at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/sample-correctness-tests.jl:75
Test Summary: | Pass Fail Total
sample-correctness-tests.jl | 153 5 158
NUTS tests with random normal | 96 96
NUTS tests with specific normal distributions | 40 40
NUTS tests with mixtures | 6 6
NUTS tests with heavier tails and skewness | 11 5 16
ERROR: LoadError: Some tests did not pass: 153 passed, 5 failed, 0 errored, 0 broken.
in expression starting at /Users/blaine/.julia/packages/DynamicHMC/n1LsS/test/runtests.jl:28
ERROR: Package DynamicHMC errored during testing

Thank you for running these. These are extreme p-values and indicate something suspicious is going on, but I cannot debug further without access to OS X.

Can you please try with Julia 1.6 on OS X and DynamicHMC 3.0.0?

I did as requested. All tests passed. There is a warning. I will wait for a response and then I plan to close this issue.

Test Summary: |
test-logging.jl | No tests
┌ Warning: (t::AbstractTransform)(x) is deprecated, use transform(t, x) instead
│ caller = rand_C(K::Int64) at common.jl:118
└ @ Main ~/.julia/packages/DynamicHMC/J48he/test/common.jl:118
INFO while testing: mixture of two normals, dimension 3
ALERT some τ = [0.164, 0.158, 0.139] < 0.15
Test Summary: | Pass Total
sample-correctness-tests.jl | 170 170
Testing DynamicHMC tests passed

The warnings are OK, some are expected (otherwise the errors would be hair-trigger).

Thanks again for investigating this.

I am closing this issue.