real::erf() inaccuracy
samhocevar opened this issue · comments
Sam Hocevar commented
The current real::erf()
implementation uses a Taylor series that does not converge and thus has precision issues.
- The affected range is [7,19].
- The approximate error is
10^(-n²/1.13)
.
That is to say, erf(8)
will have a precision error of 10^(-8²/1.13)
= 2.3e-57
, e.g. the 57th decimal will be inaccurate.
Tigeronthe commented
Hey Sam