ericmj / decimal

Arbitrary precision decimal arithmetic

Home Page:https://hexdocs.pm/decimal/

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

to_float issue with long decimal precision

tyler40AU opened this issue · comments

The following code gives 0.5, and seems to be an issue any time there are 17 or more 9s.

iex(8)> Decimal.to_float(Decimal.new("0.9999999999999999"))
0.9999999999999999
iex(9)> Decimal.to_float(Decimal.new("0.99999999999999999"))
0.5

Other numbers seem to be okay
iex(23)> Decimal.to_float(Decimal.new("0.8888888888888888888888"))
0.8888888888888888

Here is my context
iex(22)> Decimal.get_context
%Decimal.Context{flags: [], precision: 28, rounding: :half_up,
traps: [:invalid_operation, :division_by_zero]}

It seems a bit less simple than this, ex:

iex> Decimal.new("0.999999999999999990") |> Decimal.to_float
0.5

it's not 17 or more 9's, it could be any integer after there are 17 consecutive 9's

also,

iex> Decimal.new("0.99999999999999998") |> Decimal.to_float 
0.5

iex> Decimal.new("0.99999999999999995") |> Decimal.to_float
0.5