MacOS: Failed to launch GDB: Remote replied unexpectedly to 'vMustReplyEmpty'
mfamfa opened this issue · comments
I have some issues running the sample app on MacOS 12.6.2 (M1 MacBook Pro).
1. Aros config: Mostly I get the following message:
Failed to launch GDB: Remote replied unexpectedly to 'vMustReplyEmpty':
PacketSize=512;BreakpointCommands+;swbreak+;hwbreak+;QStartNoAckMode+;vContSupported+;
(from interpreter-exec console "target remote localhost:2345")
Relevant (?) output from the fs-uae.log.txt:
GDBSERVER: Waiting for connection...
connection accepted
GDBSERVER: connected
GDBSERVER: received 511 bytes: >>+$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbr<<
GDBSERVER: -> qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+
GDBSERVER: <- +
GDBSERVER: <- PacketSize=512;BreakpointCommands+;swbreak+;hwbreak+;QStartNoAckMode+;vContSupported+;
GDBSERVER: received 141 bytes: >>eak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec---+$vMustReplyEmpty#3a<<
GDBSERVER: received 1 bytes: >>+<<
GDBSERVER: disconnect
GDBSERVER: close()
[FSE] [VID] Not a valid drawable size for glViewport
[FSE] [VID] Not a valid drawable size for glViewport
[FSE] [...] Quit (maybe)
DBG: Hello debugger from Amiga!
[FSE] [VID] Not a valid drawable size for glViewport
Calling uae_quit
uae_quit
target_quit
[FAE] [ ] fs-uae shutting down, fs_emu_run returned
[FAE] [ ] state dir /Users/amor/FS-UAE/Save States/default was not removed (non-empty)
[FAE] [ ] end of main function
GDBSERVER: close()
serial: close fd=-1
Please note that during my testing the sample app actually ran 2 times (out of maybe 20 tries). When it was running I was able to break the code, single step, profile etc.
2. A500 with kickstart 1.3 config: If I try this config then FS-UAE just freezes a little while after the AmigaDos screen shows. I have to 'Force Quit' FS-UAE.
The log file is cut off at random:
[FSE] [VID] Not a valid drawable size for glViewport
[FSE] [VID] Not a valid drawable size for glViewport
[FSE] [VID] Not a valid drawable size for glViewport
GDBSERVER: Breakpoint for KPutCharX at 0xc00072 installed
GDBSERVER: Breakpoint for TRAP#7 at 0xfc0844 installed
GDBSERVER: Breakpoint for AddressError at 0xfc081a installed
GDBSERVER: Breakpoint for IllegalError
3. Lastly; How can I create a non-debug configuration that will run without attaching GDB? I assume somewhere data is written to default.fs-uae but I don't see where?
Please let me know if I can help out in any way.
Hi, I don't think that's really a GDB problem, but more of a Mac issue with FS-UAE. It seems FS-UAE quits due to problems with OpenGL? @grahambates
3. usually non-debug related stuff should be done in your own WinUAE/FS-UAE install, but you should be able to use the included FS-UAE: default.fs-uae should be in your vscode extensions directory. On windows this is %user%/.vscode/extensions/vscode-amiga-debug/bin/default.uae
remove debugging_features from there
FS-UAE is definitely quitting. The Not a valid drawable size for glViewport error may or may not be the cause. I'll compare to my log output.
I agree that running the emulator without debugging features is a good idea. You'll need to set the env var for the shared library location though:
cd ~/.vscode/extensions/bartmanabyss.amiga-debug-1.6.8/bin/darwin/fs-uae
DYLD_FALLBACK_LIBRARY_PATH=. ./fs-uaeSmall update;
- Running the sample without GDB works fine both with AROS and KS1.3 (running fs-uae from terminal with same generated default.fs-uae).
- The glViewport output is not relevant to the issue (also present when running sample without debugger).
I have a feeling is has to do with GDB/communication with GDB?
@grahambates everything runs fine on your system? Does it differ from mine?
Yeah it runs fine on my system:
Intel Macbook pro, 12.6.1
This is the log output I see:
GDBSERVER: received 472 bytes: >>+$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec<<
GDBSERVER: -> qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+
GDBSERVER: <- +
GDBSERVER: <- PacketSize=512;BreakpointCommands+;swbreak+;hwbreak+;QStartNoAckMode+;vContSupported+;
GDBSERVER: received 1 bytes: >>+<<
GDBSERVER: received 19 bytes: >>$vMustReplyEmpty#3a<<
GDBSERVER: -> vMustReplyEmpty
GDBSERVER: <- +
GDBSERVER: <-
GDBSERVER: received 20 bytes: >>+$QStartNoAckMode#b0<<
GDBSERVER: -> QStartNoAckMode
GDBSERVER: <- +
GDBSERVER: <- OK
Comparing it to yours, I'd say the line that looks dodgy is:
GDBSERVER: received 141 bytes: >>eak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec---+$vMustReplyEmpty#3a<<
It looks like a partial message.
Having the same problem on Apple M1
Do you still see the problem if you downgrade to 1.6.7?
@grahambates I am trying. Installed 1.6.7 but having some issues since FS-UAE is not static linked there is a bunch of libs that require x86 architecture. I started to uninstall my arm libs and installed the x86 versions but came to a stop with libharfbuzz that has a missing symbol required by FS-UAE:
dyld[10408]: Symbol not found: (_FT_Get_Transform)
Referenced from: '/Users/a/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/libharfbuzz.0.dylib'
I installed the latest harfbuzz 5.3.1 - maybe not the version needed?
I am also trying to reproduce the the GDB problem on my older x86 based MacBook Pro (running MacOS Catalina) but I am getting similar issues with dynamic linked libs and required missing symbols:
dyld: Symbol not found: __ZNSt3__113basic_filebufIcNS_11char_traitsIcEEE4openEPKcj
Referenced from: /Users/a/.vscode/extensions/bartmanabyss.amiga-debug-1.6.8/bin/darwin/opt/bin/m68k-amiga-elf-gdb (which was built for Mac OS X 12.0)
Expected in: /usr/lib/libc++.1.dylib in /Users/a/.vscode/extensions/bartmanabyss.amiga-debug-1.6.8/bin/darwin/opt/bin/m68k-amiga-elf-gdb
Ah yeah, I addressed that in the current build. It needs x86 versions of the dylibs because the FS-UAE binary is x86. You should be able to grab them from 1.6.8 where they are now bundled.
I was wondering if the issue is related to the new gdb version. FS-UAE hasn't really changed.
I got it running on 1.6.7 - same issue with GDB (GNU gdb (GDB) 13.0.50.20220509-git)
GDBSERVER: Waiting for connection...
connection accepted
GDBSERVER: connected
GDBSERVER: received 511 bytes: >>+$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec$qSupported:multiprocess+;swbreak+;hwbr<<
GDBSERVER: -> qSupported:multiprocess+;swbreak+;hwbreak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+
GDBSERVER: <- +
GDBSERVER: <- PacketSize=512;BreakpointCommands+;swbreak+;hwbreak+;QStartNoAckMode+;vContSupported+;
GDBSERVER: received 141 bytes: >>eak+;qRelocInsn+;fork-events+;vfork-events+;exec-events+;vContSupported+;QThreadEvents+;no-resumed+;memory-tagging+#ec---+$vMustReplyEmpty#3a<<
GDBSERVER: received 1 bytes: >>+<<
GDBSERVER: disconnect
GDBSERVER: close()
DBG: Hello debugger from Amiga!
Ok thanks for checking. It rules out the gdb change. Looks like I need an M1 Mac for testing! I might actually be able to borrow one from work.
@grahambates Please let me know if I can help out in any way; testing, debugging..
I am also available to help if needed
I tried the A1200 conf instead of AROS. Then the debugger works, but the demo program itself crashes instead :)
If it is any help, regarding:
- A500 with kickstart 1.3 config: If I try this config then FS-UAE just freezes a little while after the AmigaDos screen shows. I have to 'Force Quit' FS-UAE.
I have the same experience on my old 2013 MBP (Big Sur). It turns out that after 16 mins the sample demo is actually started and runs. Here's what the logs says:
GDBSERVER: Waiting for connection...
[FSE] [DAT] Load SairaCondensed-SemiBold.ttf
[FSE] [DAT] fsemu_data_file_path relative=SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/../Resources/Data/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/data/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Path: /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/data/SairaCondensed-SemiBold.ttf
[FSE] [FNT] Loaded SairaCondensed-SemiBold.ttf (28)
[FSE] [DAT] Load SairaCondensed-SemiBold.ttf
[FSE] [DAT] fsemu_data_file_path relative=SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/../Resources/Data/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Checking /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/data/SairaCondensed-SemiBold.ttf
[FSE] [DAT] Path: /Users/hulkhojgaard/.vscode/extensions/bartmanabyss.amiga-debug-1.6.7/bin/darwin/fs-uae/data/SairaCondensed-SemiBold.ttf
[FSE] [FNT] Loaded SairaCondensed-SemiBold.ttf (36)
[FSE] [OSM] Open on-screen menu
GDBSERVER: timed out after 1000s
DBG: Hello debugger from Amiga!
[FSE] [VID] Not a valid drawable size for glViewport
[FSE] [VID] Not a valid drawable size for glViewport
...
But, it runs very slowly, and there is no communication to VS-Code so I still have to force quit FS-UAE.
@BangKarlsen I get the same timeout after 1000 seconds. Then the demo sample runs normal and at full speed. I can mouse click out to AmigaDos and quit FS-UAE as normal.
GDBSERVER: Waiting for connection...
GDBSERVER: timed out after 1000s
DBG: Hello debugger from Amiga!
So I can confirm the two different behaviours for AROS and KS1.3. Without having a clue to what it means or if it's relevant at all.
Thanks for all the additional info. I've been trying to stay away from computers over the Xmas period but should be able to spend some time looking into this now!
I'm starting to suspect some kind of race condition. I've also seen the situation where the debugger doesn't connect and you need to force-quit, which only happens occasionally on my machine. I'll do some more poking around and see what I find.
Ok I've figured out what's going on here.
GDB retries the qSupported command, sending it multiple times as it waits for the remote program to start. The time it takes the Amiga program to start is affected by a few things including the Kickstart version and machine spec. If GDB sends the command more than three times this will exceed the 512 byte buffer in FS-UAE, causing it to trunctate and the remaining text is interpreted as part of the next command (vMustReplyEmpty).
The best way to address this would be be set the remote timeout arg to something higher than the default 2 secs when starting GDB. I'll open a PR with this fix.
@grahambates Great news! Anyway I can test it out or have to wait for new release?
@grahambates Great news! Anyway I can test it out or have to wait for new release?
Yeah you can checkout the latest master, npm install, open it in VS Code and run the project:
@grahambates I finally had time to test this.
It works - but there is a weird behaviour that after a succesful run then the next two runs will hang with the following msg:
Failed to launch GDB: localhost:2345: Operation timed out. (from interpreter-exec console "target remote localhost:2345").
This message comes after the 10 sec timeout.
Consistant over 19 runs. 1 successful then two fails, repeat.
Output from extension after failed run:
attempt to send more than one response for command launch
@mfamfa Thanks for this. I should be able to spend some time looking into this now. Just a few of questions:
- When you say it hangs, do you mean FS-UAE hangs i.e. it launches and needs a force quit?
- Can you share the FS-UAE logs from one of these failed attempts?
- Does the 'two fails' pattern still occur if you wait a couple of minutes between attempts?
I get occasional failures where FS-UAE fails to bind to the port and then hangs. It doesn't follow the same consistent pattern but I'm wondering if it could be related.
@grahambates
Yes. FS-UAE launches. Hangs in AmigaDOS and then needs a force quit. In VSCode the time out msg appears.
I have attached a log from such a session. Maybe the interesting part is this?
[UAE] [000] Switching JIT direct off!
[UAE] [000] mapped_malloc nodirect UAE Boot ROM 0x7fbcb0720000
[UAE] [000] TRAP_ENTRY = 00f021de
GDBSERVER: enabled (start_timer: 1000 trigger: :a.exe port: 2345)
GDBSERVER: close()
-----------------> STUB: get_plugin_path, size: 1000, path: debugger
-----------------> STUB: get_plugin_path, size: 1000, path: debugger\fd
my_opendir failed
GDBSERVER: listen()
GDBSERVER: bind() failed
MMAN: mapped_malloc 0x00000000 start=0x00000000 Kickstart ROM (kick)
MMAN: Flags: ROM THREADSAFE
mapped_malloc nodirect Kickstart ROM 0x7fbcb05d0000
MMAN: mapped_malloc 0x00000000 start=0x00000000 Chip memory (chip)
MMAN: Flags: RAM THREADSAFE
mapped_malloc nodirect Chip memory 0x7fbcb07d0000
MMAN: mapped_malloc 0x00000000 start=0x00c00000 Slow memory (bogo)
MMAN: Flags: RAM THREADSAFE
mapped_malloc nodirect Slow memory 0x7fbcb08d8000
ROM loader.. (<none>)
Known ROM 'KS ROM v1.3 (A500,A1000,A2000)' loaded
ROM loader end
I tried to wait a few minutes between each run and it works fine every time then. Port not freed properly?
Yeah it does look like GDBSERVER: bind() failed is the important bit and I suspect it is because the port is still in use as you suggest. This is what I'm seeing, but only on an intermittent basis. This gives me enough to go on now so hopefully I can get to the bottom of it.
One more thing to try, does it make any difference whether you quit FS-UAE directly or by pressing the 'stop debugging' button in VS Code on the previous run?
If I use the "stop debugging" in VS code then I cannot reproduce the hang (!). I can start and stop as quickly as possible without any issue.
I'm seeing the same thing. I've never been able to reproduce it when closing the emulator that way. I suspect that this is because the connection is closed before killing the process so the port isn't held open.
Now we know what the problem is I should be able to do something on either the extension or the FS-UAE side to address this.
I've updated #197 to include a fix for this. @mfamfa if you could test that branch that would be great.
@BartmanAbyss you might want to look at whether this fix is needed for WinUAE too. Basically we were already setting the SO_REUSEADDR option which should prevent this problem, but it needs to be before the call to bind().
I think this is safe to close now. Tested with latest release.