Giters

bgkillas / kalc

a complex numbers, 2d/3d graphing, arbitrary precision, vector/matrix, cli calculator with real-time output and support for units

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

0.01 is 1e-2, not 1e-3

MarkSwanson opened this issue · comments

commented

.01
1/100
1e-3

Also, if I type 1e-2 kalc gives me back 1e-3:

1e-2
1/100
1e-3

In both cases kalc should give me back 1e-2

commented

Another example:

1e-2 * 1e-2
1/10000
1e-5

The answer should be 1e-4

commented

its because '1/10^n' cant be represented in binary so my rounding system rounds it and changes it to a 1 but forgets to fix the exponent what great fun

fixed hopefully in a way that fixes all instances of this 4bd6372

commented

Glad you're havin' fun! :-) hehe