vis-lua.c:504:21: warning: implicit conversion changes value

[Don Allen]

I am posting this as an issue rather than a pull request, since I am not sure how to fix this, given that Lua is involved, about which I know just enough to be dangerous.

The code in question is the second line of this function:

static size_t checkpos(lua_State *L, int narg) {
        lua_Number n = luaL_checknumber(L, narg);
        if (n >= 0 && n <= SIZE_MAX && n == (size_t)n)
                return n;
        return luaL_argerror(L, narg, "expected position, got number");
}

Let's discuss this for 64-bit machines first. The problem is that n is a lua_Number, which is a double, which has a max significand precision of 53 bits. SIZE_MAX is 0xFFFFFFFFFFFFFFFF, obviously requiring 64 bits to represent precisely. So SIZE_MAX can't be represented precisely as a double and gets rounded up by the compiler as an implicit coercion to the next higher integer that can be represented precisely as a double, namely 0x10000000000000000. The next such integer is 0x10000000000000000+0x800. 0x800 = 2048 = 2^(64-53). So when n gets above (2^53)-1, it becomes hit or miss as to whether n can be represented precisely as a double, making the above comparison unreliable. The bigger n gets, the worse things get.

For example,

dca@giovanni:/home/dca/Software/FreeBSD/vis$ lua54
Lua 5.4.6  Copyright (C) 1994-2023 Lua.org, PUC-Rio
>  foo = (2^60)-1
> bar = foo + 1
> bar-foo
0.0
>

I think the solution is to check that

n <=(2^<significand precision>)-1

for whatever word size machine you are dealing with. This reduces the number of positions the editor can handle, which is not a problem on 64-bit machines, since (2^53)-1 is still astronomical (9,007,199,254,740,991).

It could be a problem on 32-bit machines, where the default type for lua_Number is float, which has a significand precision of 24 bits. (2^24)-1 = 16,777,215. If this is insufficient, it can be fixed by configuring Lua to use double, not float, for lua_Number. But again, since I know little about Lua or what compromises you are willing to make (e.g, adopting a least-common-denominator approach to simplify things), I'm not sure how to implement this .

[Randy Palamar]

This is specifically in reference to the following warning:

vis-lua.c:504:21: warning: implicit conversion from 'unsigned long' to 'lua_Number' (aka 'double') changes value from 18446744073709551615 to 18446744073709551616 [-Wimplicit-const-int-float-conversion]
        if (n >= 0 && n <= SIZE_MAX && n == (size_t)n)
                        ~~ ^~~~~~~~
/usr/include/bits/stdint.h:20:25: note: expanded from macro 'SIZE_MAX'
#define SIZE_MAX        UINT64_MAX
                        ^~~~~~~~~~
/usr/include/stdint.h:51:21: note: expanded from macro 'UINT64_MAX'
#define UINT64_MAX (0xffffffffffffffffu)
                    ^~~~~~~~~~~~~~~~~~~
1 warning generated.