`_₂F₁` returns wrong (and negative!) result for large parameter values
devmotion opened this issue · comments
The following example causes issues in StatsFuns:
julia> using HypergeometricFunctions
julia> _₂F₁(6041, -2495, 6042, 0.1)
-1.8601060304844724e86According to Mathematica the result should be around 7.169e-115 (see, e.g., WolframAlpha).
That's a tough one since it's so ill-conditioned!
Even a switch to BigFloat doesn't give the true result with 256 bits.
julia> for d in (64, 128, 256, 512, 1024, 2048)
setprecision(d)
println("With $d bits: ", _₂F₁(big(6041), big(-2495), big(6042), big"0.1"))
end
With 64 bits: -8.99485706711407646963e+82
With 128 bits: -3.768364622629911259387373377831295071159e+63
With 256 bits: 7.643085578305798769538104224081012613229772525309794740110258747017712951371203e+24
With 512 bits: -4.4264798677310339256173502040188038311890682446912080069351277321056166885881673669408836469214270669989163506221614309672646227499147896742387972120237901e-53
With 1024 bits: 7.169000864829757581421226521118341577071490491462656813054559391246650807976143869415974650424976180664819945767536481261295942933662658520034114786078310397596431648363554816973760905648122364023602984747002471585997491094811128251440473364079854567611045461854182756393668302607360226003994524772703104486931e-115
With 2048 bits: 7.16900086482975758142122652111834157707149049146265681305455939124665080797614386941597465045815641567402422575960186149900525065034794034245254231059789333329885896628183155673583792646799563492848650702878239420924841585546950533124023812208251395253944943499468495262977909104723971416703356838132623167890376330260360041712490878775613493723688176758406527219283758417803787986529538500458406631271909953012349496024826178457978964413609239990864787077478180426443391470025827655872136792733048071405467264884944257127728627481742532104296302397542778585748814649159904153814745638294080221721631746185393466297522e-115
Please be advised this package is not symbolic in any way. But it does support some rational approximations that turn out to be a bit better:
julia> for d in (64, 128, 256, 512, 1024, 2048)
setprecision(d)
println("With $d bits: ", HypergeometricFunctions.pFqweniger([big(6041), big(-2495)], [big(6042)], big"0.1"))
end
With 64 bits: 7.33313041116102000485e-18
With 128 bits: -1.315752681882871685499142091512016853179e-37
With 256 bits: -5.88011503082003285608549269073787834531678494727957553278509703250723096730728e-76
With 512 bits: 7.16900086482975758142122652111834157693588006812966718922489682313836019469317922584726132521494747133919436666829291677405009314509901237278755322399783044e-115
With 1024 bits: 7.169000864829757581421226521118341577071490491462656813054559391246650807976143869415974650458156415674024225759601861499005250650347940342452542310597893333298858966281831556735837926467995641393151209833000746729910324899663179564716640497352333266144750526037598526100148693723982529553736819701265169463323e-115
With 2048 bits: 7.16900086482975758142122652111834157707149049146265681305455939124665080797614386941597465045815641567402422575960186149900525065034794034245254231059789333329885896628183155673583792646799563492848650702878239420924841585546950533124023812208251395253944943499468495262977909104723971416703356838132623167890376330260360041712490878775613493723688176758406527219283758417803787986529538500458406631294807307181301996737860383586928193338206438102619894396979075216057868001561137227723845183386807553903964830789880785321005000975898230383185412057088684678130334133157201835739033936199142693848081407373683764397366e-115
Looking at the betalogcdf implementation, maybe there's a reason to use https://dlmf.nist.gov/8.17#E8 instead
julia> a = 6041
6041
julia> b = 2496
2496
julia> x = 0.1
0.1
julia> (1-x)^b * HypergeometricFunctions.pFqweniger([1, a+b], [a+1], x)
7.169000864830204e-115I think with a,b>0, it's true that this is positive, something that can't be said about the terminating form with alternating coefficients.
Probably this case converts the DLMF's E8 into E7 for the _₂F₁, but the rational algorithm isn't cased out.
Yes, maybe we should use a different algorithm in StatsFuns - in particular since one problem with these user-provided parameter values is that it leads to a DomainError due to the negative result in HypergeometricFunctions. Thanks for looking into the issue!
I think with a,b>0, it's true that this is positive, something that can't be said about the terminating form with alternating coefficients.
Do you know any (immediate) disadvantages or possible problems with this alternative approach based on pFqweniger? I'm not familiar with the different algorithms in HypergeometricFunctions, are there e.g. any performance difference or possible numerical issues for certain parameter ranges?
But it does support some rational approximations that turn out to be a bit better:
Probably this case converts the DLMF's E8 into E7 for the _₂F₁, but the rational algorithm isn't cased out.
I think, from the perspective of a downstream user it would be easiest if HypergeometricFunctions would automatically use the algorithm with the best trade-off between performance and accuracy even for such hard parameter choices. I would also like to avoid calling internal functions such as pFqweniger in StatsFuns.
(BTW while these parameter values may seem a bit unlikely it's easy to end up with such values eg in a simple Bayesian coinflip example: https://turing.ml/dev/tutorials/00-introduction/)
In the general setting, it's not at all obvious to me what's best: if one of the top parameters is a real negative integer, the series is terminating. If all the parameters and the argument are positive, the series is positive. How can one even begin to weigh these two properties?
Maybe I could add a method positive₂F₁ just for the important scenario of a CDF: real argument clamped between 0 and 1, inclusive, and positive parameters?
Just tagged a new version with this exported function positive₂F₁ with argument support that includes the use for the beta log CDF. Let me know if it works for StatsFuns with https://dlmf.nist.gov/8.17#E8
Wow, thanks a lot for the quick fix! So far it seems to solve the issues in StatsFuns: JuliaStats/StatsFuns.jl#144
Hmm, it seems to cause downstream issues for some non-finite special cases: https://github.com/JuliaStats/StatsFuns.jl/runs/6414587037?check_suite_focus=true#step%3A6%3A487 I'll check how we can fix them. The original problem is resolved at least 🙂
Does it have to do with argument unity?
The problem are non-finite parameter values, which can occur since we reduce computation of logccdf(Binomial(n, p), k) to logcdf(Beta(max(0, k + 1), max(0, n - k)), p). We want logccdf(Binomial(n, p), Inf) = log(0.0) = -Inf. This worked with E7 since then only one of the terms in the branch in Edit: I was using the wrong formula (not in the PR, just this comment), it seems the betalogcdf (the one that calls HypergeometricFunctions) is non-finite, but now the call of positive₂F₁ also returns a non-finite value with different sign, so we end up with NaN.NaN arises directly from the call to positive₂F₁:
As a concrete example, we have
julia> binomlogccdf(5.0, 0.5, Inf) # should be -Inf
NaN
julia> betalogcdf(Inf, 0.0, 0.5) # called by `binomlogccdf`
NaN
julia> positive₂F₁(Inf, 1.0, Inf, 0.5) # non-finite term introduced in the PR
NaN
julia> _₂F₁(Inf, 1.0, Inf, 0.5) # finite term in the current version
2.0Maybe we have to handle this special case separately.
Maybe we have to handle this special case separately.
On the other hand, maybe this should be fixed in HypergeometricFunctions since it worked before and we should still get
julia> positive₂F₁(Inf, 1.0, Inf, 0.5)
2.0as before. I guess maybe a branch in the definition of positive₂F₁ would be sufficient?
We can do that: when a top and bottom parameter are equal, we remove them both
HypergeometricFunctions.jl/src/gauss.jl
Lines 9 to 12 in 274185d
Ok, so I changed it again 55bd4f2. Instead of positive2F1, there's a method keyword which can guarantee positivity (though that certainly comes at a computational cost for z near 1) _₂F₁(a, b, c, z; method = :positive).
This lets the code reuse all the special cases based on parameter values (including the removal of two equal values in top/bottom) and I also added an argument unity case.
Thank you! It seems this fixed the downstream tests in StatsFuns. I'll close this issue.