IBM360Converter converts to IEEE NaN for some values
lpartington opened this issue · comments
IBM360Converter compares the IBM float value (as an integer) with IEMAXIB, returning IEEEMAX (+Inf) if the value is larger than IEMAXIB.
IEMAXIB is defined as:
public static final int IEMAXIB = 0x611FFFFF;
The IBM value 0x61100000 corresponds to IEEE +Inf (0x7F800000, signifying exponent FF and mantissa 0). IBM values between 0x61100001 up to 0x611FFFFF get converted to IEEE values in the range 0x7F800008 - 0x7FFFFFF8. These are NaN values, where exponent = FF and mantissa is not 0. Then IBM values from 0x61200000 upwards are converted to +Inf again (due to the IEMAXIB test).
It would appear that IEMAXIB should be 0x61100000 rather than 0x611FFFFF so that all values above 0x61100000 are converted to +Inf.
If 0x61100000 is used, then the comparison could be changed to >= IEMAXIB for clarity (although since val == IEMAXIB yields +inf anyway, it doesn't really matter). Alternatively, > 0x610FFFFF could be used.