Tmpfile on Windows will often not work, cascades into `PARSE_STACK` error on fix.

Question

Tmpfile on Windows will often not work, cascades into `PARSE_STACK` error on fix.

cr1901 opened this issue a year ago · comments

Starting from ee3329c, we use tmpfile to create temporary files. When running tests, we will get output similar to this on Windows, where I added a perror call:

William@DESKTOP-3H1DSBV MINGW64 ~/Projects/legacy/wla-dx
$ ./run_tests.sh
Building byte_tester...
make ; cp byte_tester ../binaries
make[1]: Entering directory '/home/William/Projects/legacy/wla-dx/byte_tester'
gcc  main.o -o byte_tester
make[1]: Leaving directory '/home/William/Projects/legacy/wla-dx/byte_tester'

########################################################################
WARNING: Valgrind is not installed so we cannot perform memory checks...
########################################################################

Running tests...


rm -f main.o more.o core *~ *.lst linked.rom linked.sym
wla-6502 -i -o main.o main.s
Permission denied
MAIN: Error creating a tmp file for WLA's internal data stream.
make: *** [makefile:15: main.o] Error 1

########
FAILURE!
########
Test "after_test/" of platform "6502/" failed.

perror returns Permission denied. It turns out that for historical reasons, Windows writes tmpfiles to the root directory, which you often don't have permission to write to. Microsoft docs recommends using tmpnam and fopen. I've supplied a partial patch here:

diff --git a/main.c b/main.c
index 782e99a..33348ee 100644
--- a/main.c
+++ b/main.c
@@ -89,6 +89,10 @@ static int s_deletable_structures_max = 0, s_deletable_structures_count = 0;
 extern char *g_sdsctag_name_str, *g_sdsctag_notes_str, *g_sdsctag_author_str;
 #endif
 
+#ifdef WIN32
+  static char * s_tmpfile_name;
+#endif
+
 PROFILE_GLOBALS();
 
 
@@ -526,6 +530,11 @@ static void _procedures_at_exit(void) {
   if (g_file_out_ptr != NULL) {
     fclose(g_file_out_ptr);
     g_file_out_ptr = NULL;
+
+#ifdef WIN32
+    remove(s_tmpfile_name);
+    s_tmpfile_name = NULL;
+#endif
   }
 
   free(g_macro_stack);
@@ -951,7 +960,28 @@ int main(int argc, char *argv[]) {
     return 1;
   }
 
+#ifdef WIN32
+  /* tmpfile uses root of drive on Windows, which is probably not-writable. */
+  s_tmpfile_name = tmpnam(NULL);
+
+  if (s_tmpfile_name == NULL) {
+    fprintf(stderr, "MAIN: Error creating a tmp filename for WLA's internal data stream.\n");
+    return 1;
+  }
+
+  /* Per: https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/tempnam-wtempnam-tmpnam-wtmpnam?view=msvc-170&redirectedfrom=MSDN
+  When a file name is prepended with a backslash and no path information,
+  such as \fname21, it indicates that the name is valid for the current working
+  directory. */
+  if(s_tmpfile_name[0] == '\\') {
+    s_tmpfile_name = s_tmpfile_name + 1;
+  }
+
+  g_file_out_ptr = fopen(s_tmpfile_name, "wb");
+  fprintf(stderr, "%s\n", s_tmpfile_name);
+#else
   g_file_out_ptr = tmpfile();
+#endif
   if (g_file_out_ptr == NULL) {
     fprintf(stderr, "MAIN: Error creating a tmp file for WLA's internal data stream.\n");
     return 1;

Unfortunately, after applying this patch, the link fails with a PARSE_STACK error:

$ ./run_tests.sh
Building byte_tester...
make ; cp byte_tester ../binaries
make[1]: Entering directory '/home/William/Projects/legacy/wla-dx/byte_tester'
gcc  main.o -o byte_tester
make[1]: Leaving directory '/home/William/Projects/legacy/wla-dx/byte_tester'

########################################################################
WARNING: Valgrind is not installed so we cannot perform memory checks...
########################################################################

Running tests...


rm -f main.o more.o core *~ *.lst linked.rom linked.sym
wla-6502 -i -o main.o main.s
shm0.
wla-6502 -i -o more.o more.s
s754.
wlalink -v -i -s linkfile linked.rom
more.o: more.s:78: PARSE_STACK: Unresolved reference to "label_2X".
make: *** [makefile:12: all] Error 1

########
FAILURE!
########
Test "after_test/" of platform "6502/" failed.

Have any idea on the follow-up error? I've attached my after_test directory with assembler output. Patch should be applied against: 45fc096.

Ville Helin · Answer 1 · Thu Jul 20 2023 19:26:23 GMT+0800 (China Standard Time)

I'm using Cygwin under Windows 11 myself to develop WLA DX and there everything works as normally. But I see you are using MinGW64? I'll try to install that and recreate the problem...

Ville Helin · Answer 2 · Thu Jul 20 2023 19:40:32 GMT+0800 (China Standard Time)

Ok, it seems I don't know how to use MinGW

What is this shell? I think I need more info about how to set up the same environment that you have...

Ville Helin · Answer 3 · Thu Jul 20 2023 19:55:14 GMT+0800 (China Standard Time)

I ran your after_test under Cygwin:

First without assembling anything, just linking -> got the same error as you got. After make clean it worked just fine. I assume your version of wla-6502 is somehow broken...

Ville Helin · Answer 4 · Thu Jul 20 2023 20:20:39 GMT+0800 (China Standard Time)

I added your patch with small mods, but now we have your PARSE_STACK issue...

Ville Helin · Answer 5 · Thu Jul 20 2023 20:33:22 GMT+0800 (China Standard Time)

It seems that after_test assembled using MinGW makes the objects a lot smaller

Here's assembled with Cygwin:

Ville Helin · Answer 6 · Thu Jul 20 2023 20:40:44 GMT+0800 (China Standard Time)

I think it was this. I changed it to "w+" as we are also reading from that file and Windows x86 Azure Pipelines task completed successfully.

If you try out the latest sources, does it work now?

William D. Jones · Answer 7 · Thu Jul 20 2023 22:16:59 GMT+0800 (China Standard Time)

Sorry, I was asleep. You were busy while I was asleep :D!

What is this shell? I think I need more info about how to set up the same environment that you have...

I'm using MSYS2 to get a bash shell and the mingw-w64 compilers. Looks like you've installed just the mingw-w64 compilers.

If you try out the latest sources, does it work now?

Ugh, the documentation even says tempfiles use +!!

The temporary file is opened in w+b (binary read/write) mode.

I thought it meant w and b, and forgot the + can also be in the middle. I added the b locally as well to be extra safe (does not affect what happens below).

Now we get further into the tests, but now we fail at one of the 8080 tests:

./run_tests.sh
[30/30] Linking C executable binaries\wla-huc6280.exe
Building byte_tester...
make ; cp byte_tester ../binaries
make[1]: Entering directory '/home/William/Projects/legacy/wla-dx/byte_tester'
gcc  main.o -o byte_tester
make[1]: Leaving directory '/home/William/Projects/legacy/wla-dx/byte_tester'

########################################################################
WARNING: Valgrind is not installed so we cannot perform memory checks...
########################################################################

Running tests...
.............................................................

rm -f core *~ linked.rom linked.sym *.o
wla-8080 -M -o main.o main.s
main.s:138: INCLUDE_FILE: Error creating a tmp file for "includeXYZ3.s"!
main.s:138: ERROR: Couldn't parse ".include".
make: *** [makefile:11: main.o] Error 1

########
FAILURE!
########
Test "makefile_generation/" of platform "8080/" failed.

Hmm...

Ville Helin · Answer 8 · Thu Jul 20 2023 22:31:24 GMT+0800 (China Standard Time)

If you rerun the tests do you still get the same error in the same place? I'm wondering if that is now something that happens randomly once in a while as the Windows x86 Azure Pipelines task completed successfully...

Ville Helin · Answer 9 · Thu Jul 20 2023 22:40:38 GMT+0800 (China Standard Time)

Could you debug why you get this?

Like fprintf() the tmp fle name or step it in a debugger?

Ville Helin · Answer 10 · Thu Jul 20 2023 23:15:46 GMT+0800 (China Standard Time)

I ran the x86 Azure Pipelines task

two times again, manually, and both times it succeeded.

Ville Helin · Answer 11 · Thu Jul 20 2023 23:49:43 GMT+0800 (China Standard Time)

Ok, I installed MSYS2 and all tests passed. Perhaps I'm using it differently than you?

William D. Jones · Answer 12 · Thu Jul 20 2023 23:50:23 GMT+0800 (China Standard Time)

Okay something is clearly weird on my end then. I'll debug now.

William D. Jones · Answer 13 · Fri Jul 21 2023 00:15:45 GMT+0800 (China Standard Time)

I think I found the culprit: https://github.com/vhelin/wla-dx/blob/master/include.c#L140

There's other tmpfile calls in the source. Perhaps spin out to a tmpfile.c source file? I can try to take a stab at this.

Also:

Should this line be succeeding in tests?

.include { "include{XYZ_STR}3.s" }

The source file in tests/8080/makefile_generation is called includeXYZ.s, not includeXYZ3.s.

Ville Helin · Answer 14 · Fri Jul 21 2023 00:26:41 GMT+0800 (China Standard Time)

It seems tmpfile() is called in three .c files! I had totally forgotten this...

Making tmpfile.c would require changes to Visual Studio Solution and makefiles etc., perhaps just make the tmpfile etc. creation function public and place it in main.c?

Ville Helin · Answer 15 · Fri Jul 21 2023 00:30:32 GMT+0800 (China Standard Time)

Also:

Should this line be succeeding in tests?
.include { "include{XYZ_STR}3.s" }
The source file in tests/8080/makefile_generation is called includeXYZ.s, not includeXYZ3.s.

You are correct, perhaps that's a typo, but as the "makefile_generation" tests just tests that makefile generation doesn't exit with an error it works even though the generated makefile is a bit broken...

William D. Jones · Answer 16 · Fri Jul 21 2023 00:36:43 GMT+0800 (China Standard Time)

perhaps just make the tmpfile etc. creation function public and place it in main.c?

I can do this, and bypass the need for a main.h header by the tmpfile function public in each file that uses it.

The "wonderful" thing about tmpfile being broken is that the guarantees we get with tmpfiles being cleaned up no longer apply with the tmpnam/fopen dance. So I have to register cleanup when wla-* exits and I'm required to keep the temporary filenames around. Hence why I wanted to spin it out into its own file. But anyways, this shouldn't be that much code, so gimme a few mins.

Ville Helin · Answer 17 · Fri Jul 21 2023 00:38:39 GMT+0800 (China Standard Time)

Well, if you want to update all the Visual Studio Solutions and historical makefiles via editing the makefile generator etc., feel free to add a new .c file to the project. :)