float/double fails in ut_xrange.c tests
devel-chm opened this issue · comments
There were 12 FAILs in the XRange tests which appear to
be the result of floating point error in calculating the number
of counts inthe range. 0.1 is not exact in floating point.
In XRange.h the code used DBL_EPSILON. I edited to use
FLT_EPSILON instead and all tests PASS. I'm not familiar
enough with the COS code base yet to determine if thios is the
correct fix.
On my system FLT_EPSILON is about 1.2e-07 so
using that instead of DBL_EPSILON should work for
ranges up to 8.4e6 or so. Given that the error would
be off by less than 1 in that range---and only because
the number are not exactly representable in floating
point format (e.g. 0.1 base 10 is 0.0001100 base 2
with the last 4 bits repeating indefinitely), it might be
a reasonable "fix"