64bit build using MSVC (Visual Studio 2022) fails
pm100 opened this issue · comments
Many fatals. Two primary causes
- treating the difference of two pointers as an int
12>C:\work\forks\cc65\src\ca65\instr.c(1892,32): warning C4244: 'return': conversion from '__int64' to 'int', possible loss of data - treating size_t as an int
12>C:\work\forks\cc65\src\ca65\objfile.c(291,1): warning C4267: 'initializing': conversion from 'size_t' to 'unsigned int', possible loss of data
These are fatals due to the 'treat warnings as errors' flags.
I have not yet tried building the libraries
I am happy to fix
of course the libraries are built with cc65 and ca65 so not relevant - duh.
I wonder what MSVC is doing there - i don't see anything like this with GCC
In msvc Size_t is 64 bit. Int and unsigned are 32
https://devblogs.microsoft.com/oldnewthing/20050131-00/?p=36563
Ah i remember now... in GCC int is 64bit.
(Their example is hilarious - like it's not the programmers fault when he relies on the size of types :D)
I have it all cleaned up. Now I need to fix the make files in order to test it all. The makefiles try to use the local build in the same tree (especially good for tests!) But it fails on windows and falls back to the path installed version
The makefiles try to use the local build in the same tree (especially good for tests!) But it fails on windows and falls back to the path installed version
That sounds wrong to me, are you sure that is what happens?
In any case, please make buildsystem fixes a separate PR :)
If you want GCC to warn about assigning 64-bit values to 32-bit types, you can use -Wconversion, but be warned: that also enables other warnings about precision loss, like assigning int to uint8_t and warns about assigning signed to unsigned and vice versa (-Wsign-conversion). A quick check with GCC 12.2 results in 2693 warnings =)
With Clang you can use -Wshorten-64-to-32 to warn specifically about losing precision when assigning 64-bit values to 32-bit types. Using Clang 16 with that flag results in 746 warnings when building cc65.
@Compyx i think the issue here is that in gcc 64 bit sizeof(void*) == sizeof(long), so there is not shortening . Thats not the case in MSVC
@mrdudz. I have spent a long long time trying to do the non cast option (ie the opposite of my original PR). It is horrible. Basically everything gets touched. I started using size_t everywhere, but realized I was using a signed type in place of an unsigned type in many places - in theory this is okish but its hard to tell down in the guts of sim64 and cc65 code generator. So I stopped that after a lot of work. I tried to stop the changes spreading (even ignoring the sign issues) too far but its impossible if casts are not allowed.
Using size_t for every signed integer and uintptr_t for every unsigned integer is the correct thing to do - this guarantees that sizeof(all integer types) == sizeof(void*) .
The problem with that is for a start it looks really weird. I could define types like cc65_int and cc65_uint (ignore the names) to make it look less weird.
But most importantly people not working on msvc64 bit builds will easily break things by using int or unsigned long or whatever, rather than the 'correct' types. To prevent this you would need to add a 64 bit msvc build to the CI workload. This is probably a thing that should be done anyway. But people will not be happy because they will have to install msvc and windows in order to check that they have fixed any issue they created, otherwise they have to keep committing to github and waiting for the CI to run.
I really suggest that you reconsider the original PR, it confines the changes to the bare minimum.
I can macroize the casts so that they include range check that will fast fail.
Using size_t for every signed integer and uintptr_t for every unsigned integer is the correct thing to do - this guarantees that sizeof(all integer types) == sizeof(void*) .
Sorry, no. This is a shotgun "solution" and certainly not the correct thing to do. Also size_t is unsigned (ssize_t is signed, but POSIX only) - which explains why you run into problems with it. In any case, as said before, the correct solution is to use the proper types at the proper places - not making every int the same type.
But most importantly people not working on msvc64 bit builds will easily break things by using int or unsigned long or whatever, rather than the 'correct' types. To prevent this you would need to add a 64 bit msvc build to the CI workload. This is probably a thing that should be done anyway. But people will not be happy because they will have to install msvc and windows in order to check that they have fixed any issue they created, otherwise they have to keep committing to github and waiting for the CI to run.
That's exactly why we have the CI. It shouldn't be much of an issue anyway.
TBC when I said
sizeof(all integer types) == sizeof(void*)
I mean
sizeof(int)==sizeof(unsigned)==sizeof(unsigned long) == sizeof(long)==sizeof(void*) == sizeof(size_t)
I did not mean every integral type . Its these types, int/long/unsigned, that need to be changed.
The current code base definitely assumes this (not unreasonably I might add) and is why it fails to compile on 64 bit msvc. So forcing this to be true is not a shotgun solution, its the only solution if no sizing down casts are to be used.
The assumptions are:
- that the return value from strlen is the same size as an int/long/unsigned
- that the result of subtracting 2 pointers is the same size as an int/long/unsigned
I was not doing this via edit/replace (maybe why you said 'shotgun'). I was making the first change (starting at the places that originally failed - strlen and p1-p2 in the places of the original PR) and see what doesnt compile, fixing those errors then following what then breaks, repeat, repeat...etc. A long tedious process that diverges a lot but then slowly converges. But one that minimizes the code changes. I ran into a few places where it was not clear if the changes were correct (fiddly things in the code generators, and sim65), which I why I stopped
Anyway TIL I was wrong about size_t, somehow I had it in my mind that it was signed. Its unsigned, my bad
ssize_t should not really be used for the signed equivalent for 2 reasons
- its not C standard
- its range guarantee is [-1, SSIZE_MAX] (I am sure that its better than that on real implementations, but even so)
So it would need to be size_t for unsigned things and ptrdiff_t or intptr_t for signed things. I think I would prefer to conditionally typedef something like 'wide_int' 'wide_long' just because it looks so strange to see types with 'ptr' in their name being used for lengths and counters etc.
I will have another go at it (i am retired so I have the bandwidth) . I already started from scratch on it twice, I am getting good at it now, plus I am getting to know the code base really well.
So it would need to be size_t for unsigned things and ptrdiff_t or intptr_t for signed things.
Again: no, this is not the right thing to do. The right thing to do is to use the types that are meant for the purpose they are used in. ptrdiff_t is used for the difference between two pointers. no more no less. intptr_t is used for an integer that is holding the value of a pointer, no more no less. Abusing those types for random things is the shotgun solution.
So what should the return type be for a function that returns the difference of two pointers that the caller treats as a length ? The caller stores it in a long
That can't be answered in a general way, obviously. It depends on what the function is defined to return.
Well I have to change it. At the moment it's declared to return an unsigned. And the caller stores it in an unsigned (I said long above, I meant unsigned, point was it's not ptrdidf_t). If you want a specific instance look at my pr
You'll have to look at the comments, and perhaps follow the code, and find out what the original author intended that function to return. if its the difference between two pointers, ptrdiff_t is the correct thing. If not, then it's something else. It's certainly not a trivial task nor mechanical change. (ie by "defined" i am not referring to the functions current signature)
Reset. If I have to change an inderminate integral type to a deterministic one. If unsigned I will use size_t. If signed I have 2 choices. Ptrdiif or intptr. Almost all the lengths in most structs have either come from pointer diff calculations or strlens. Sbstr certainly needs to be changed. So what do you want the type of length in sbstr to be? I will do whatever you say
If unsigned I will use size_t. If signed I have 2 choices. Ptrdiif or intptr.
This is going in circles. Those types have specific applications. Changing types the way you say is just wrong.
@mrdudz - did some further research about what errors are detected by what compilers. If I turn on the gcc error detection flags (-Wconversion -Wno-sign-conversion) so that it picks up the same issues that are triggering the MSVC warnings. I get 1804 warnings. Most are the same as the MSVC, some are different because of the different size rules between the two compilers. Ie this
ar65/library.c:141:13: warning: conversion from ‘long unsigned int’ to ‘unsigned int’ may change value [-Wconversion]
141 | Count = ReadVar (Lib);
is producing a warning in gcc 64 bit because long and int have different sizes, but they have the same size in MSVC 64 bit so no warning
Whereas this produces a warning everywhere
common/strbuf.h: In function ‘SB_CopyStr’:
common/strbuf.h:317:28: warning: conversion from ‘size_t’ {aka ‘long unsigned int’} to ‘unsigned int’ may change value [-Wconversion]
317 | SB_CopyBuf (Target, S, strlen (S));
because size_t and int / unsigned int have different sizes in both compilers
also this one
ca65/instr.c:1892:19: warning: conversion from ‘long int’ to ‘int’ may change value [-Wconversion]
1892 | return ID - InsTab->Ins;
| ~~~^~~~~~~~~~~~~
ie treating the difference between 2 64 bit pointers as a 32bit value.
These 2 are the errors that I fixed in my original PR.
I note that the gcc compiler Makefile flags are very aggressive, clearly with the intent to be as clean as possible (-Wall, -Wextra -Wno-char-subscripts) but those flags do not detect the errors that MSVC is complaining about (this was a surprise to me). Really the two sets of error detection should be the same: unsigned x = strlen(s); on a 64bit build should be consistently flagged or not since the error is identical (forcing a 64bit size_t into a 32bit unsigned)
I can fix the code so that it passes gcc and MSVC with these errors corrected and full error detection enabled.
Or I can change the MSVC setup so that it doesnt complain about the errors that gcc is ignoring
I can fix the code so that it passes gcc and MSVC with these errors corrected and full error detection enabled.
Or I can change the MSVC setup so that it doesnt complain about the errors that gcc is ignoring
What i would do... is:
a) change MSVC setup so it specifically ignores those warnings (but not any others) (This is trivial and this will not break anything.) as a temporary solution. Perhaps add a GHA for 64bit MSVC build too.
b) explicitly enable all warnings related to sign- and type-conversion for GCC and for CLANG compilation (sometimes one of those catches a problem that the other doesn't). And then very carefully try to fix all of them, for 32bit and 64bit GCC and CLANG builds. (*)
c) once that works, fix the MSVC build (it will still complain about something... :))
(*) and as said, this doesn't mean changing random things to size_t - it means carefully inspecting the code, and picking the right types for the right things. It shouldn't have to involve using fixed-size types (stdint), and typecasts should be reduced to an absolute minimum (as said before, typecast are very often just wrong and hiding errors). Also ifdefs that check for MSVC or GCC are a no-go, we really don't want that - just like using custom types as suggested before.
That said, i'd expect this to be a long and tedious process, involving quite a bit of testing and review (not only from me). Last time i did this, for an even smaller code base, it took several months of fulltime work until the (hopefully) optimal clean solution was ready.
a) OK . I assume GHA means Github Action
b) OK
c) OK
(*) At no point whatsoever was I suggesting randomly doing anything. You must have misunderstood a comment by me somewhere along the way. I was going through by hand one at a time assessing what to do. There was no bulk edit replace, there was no hiring of monkeys ... :-) It is indeed a long tedious process, thats fine, I am getting good at it having made 2 false starts at it.
The only point I would make is that some changes are a matter of opinion . (the strlen / size_t one is fine)
A concrete example is that instr.c error above. Some code paths return -1 , some return a pointer difference. The function itself is declared to return int. The value returned is ultimately used as an index into instruction tables.
So what should the function be declared as returning - the caller treats this function as a lookup in a table, that either returns -1 or the actual index
Honestly I would cast the calculation to an int. No instruction table is going to have > 2^32-1 entries. And if the code was written juggling counters and indexes rather than juggling pointers we would not be debating it, it would return an int. If gcc had reported this error with the current make flags I am sure thats what the code would now say, this error has nothing to do with MSVC weird lengths, its just that MSVC reported the error and gcc did not
If down casting is banned then it has to be something the same size as ptrdiff_t:
- it cant be size_t because thats not signed.
- It could be ssize_t but I really dont like that but could be persuaded.
- long long is certainly possible, the downside with that is that its 64 bits on a 32 bit platform.
- intptr_t really should not be used since it is not an integer representation of a pointer.
So really it would have to be ptrdiff_t which leaks implementation to the caller, plus it ripples through the code base. For example another code path (FuncIsMnemonic in expr.c) calling that same instr function stores the result in the expression tree as a long - right here
union {
long IVal; /* If this is a int value */ <==================right here
struct SymEntry* Sym; /* If this is a symbol */
unsigned SecNum; /* If this is a section and Obj != 0 */
unsigned ImpNum; /* If this is an import and Obj != 0 */
struct Import* Imp; /* If this is an import and Obj == 0 */
struct MemoryArea* Mem; /* If this is a memory area */
struct Segment* Seg; /* If this is a segment */
struct Section* Sec; /* If this is a section and Obj == 0 */
} V;
now that would have to be ptrdiff_t (because long is too small on 64bit MSVC) - thats very odd looking. The comment says its an int
And as a reminder: that code (silently converting the difference between to 64 bit pointers to a 32 bit integer) has been working fine for all the time there has been a 64 bit version. Casting it to an int would formally bless the accidental behavior thats shipping now
------- stdint.h--------
The only place where I think fixed size types should be used is for things that explicitly seem (to me ) to demand it. For example Read8 should surely return uint8_t. But thats a different issue and a different set of PR.