Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IBM360Converter converts to IEEE NaN for some values #555

Open
lpartington opened this issue Apr 11, 2023 · 1 comment
Open

IBM360Converter converts to IEEE NaN for some values #555

lpartington opened this issue Apr 11, 2023 · 1 comment

Comments

@lpartington
Copy link

IBM360Converter compares the IBM float value (as an integer) with IEMAXIB, returning IEEEMAX (+Inf) if the value is larger than IEMAXIB.
IEMAXIB is defined as:
public static final int IEMAXIB = 0x611FFFFF;
The IBM value 0x61100000 corresponds to IEEE +Inf (0x7F800000, signifying exponent FF and mantissa 0). IBM values between 0x61100001 up to 0x611FFFFF get converted to IEEE values in the range 0x7F800008 - 0x7FFFFFF8. These are NaN values, where exponent = FF and mantissa is not 0. Then IBM values from 0x61200000 upwards are converted to +Inf again (due to the IEMAXIB test).
It would appear that IEMAXIB should be 0x61100000 rather than 0x611FFFFF so that all values above 0x61100000 are converted to +Inf.

@lpartington
Copy link
Author

If 0x61100000 is used, then the comparison could be changed to >= IEMAXIB for clarity (although since val == IEMAXIB yields +inf anyway, it doesn't really matter). Alternatively, > 0x610FFFFF could be used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant