to_float issue with long decimal precision
tyler40AU opened this issue · comments
The following code gives 0.5, and seems to be an issue any time there are 17 or more 9s.
iex(8)> Decimal.to_float(Decimal.new("0.9999999999999999"))
0.9999999999999999
iex(9)> Decimal.to_float(Decimal.new("0.99999999999999999"))
0.5
Other numbers seem to be okay
iex(23)> Decimal.to_float(Decimal.new("0.8888888888888888888888"))
0.8888888888888888
Here is my context
iex(22)> Decimal.get_context
%Decimal.Context{flags: [], precision: 28, rounding: :half_up,
traps: [:invalid_operation, :division_by_zero]}
It seems a bit less simple than this, ex:
iex> Decimal.new("0.999999999999999990") |> Decimal.to_float
0.5
it's not 17 or more 9's, it could be any integer after there are 17 consecutive 9's
also,
iex> Decimal.new("0.99999999999999998") |> Decimal.to_float
0.5
iex> Decimal.new("0.99999999999999995") |> Decimal.to_float
0.5
Fixed in #83.