Repository navigation
Dropped 'gdb' connection #348
Description
Environment:
OS: Windows 10, 11, MacOS (tried in all)
Visual Studio Code
Native Debug version v0.26.0
OpenOCD session running in background
Hardware ESP32
JTAG hardware ESP-prog
Issue:
My debugging environment stops working after updating to v0.26.0. When starting a session openocd reports gdb session as dropped.
After reverting to v0.25.1 it works as usual.
Copy of launch.json
{
"version": "0.2.0",
"configurations": [
{
"type": "gdb",
"request": "launch",
"name": "Flash and Debug Jlink",
"target": "./build/ninja-fiw.elf",
"cwd": "${workspaceFolder}",
"gdbpath": "xtensa-esp32-elf-gdb",
"autorun": [
"target remote :3333",
"mon reset halt",
"flushregs",
"thb app_main",
"c"
]
}
]
}
If submitting a bug please make sure
- If you are using gdb
-
gdb --version>= 7.7.1 - it works on the command line with
gdb -
cwdandtargetare properly set
-
- If you are using lldb
-
lldb --version>= 3.7.1 - it works on the command line with
lldb-mi -
cwdandtargetare properly set
-
Screenshots are helpful but not required
Activity
GitMensch commentedon Apr 26, 2022
Thanks for the report, I guess that's an issue with the new startup.
Please report
xtensa-esp32-elf-gdb --versionand do a run with added"showDevDebugOutput": true,"printCalls": true,this way we can see what the difference is.Other than this you possibly could try to configure remote debugging instead of the launch option, see https://github.com/WebFreak001/code-debug#using-gdbserver-for-remote-debugging-gdb-only for details.
ericashede commentedon Apr 26, 2022
xtensa version:
Debug console output:
WebFreak001 commentedon Apr 26, 2022
hm this seems like weird output:
can you also run again with downgradede extension where it works and send the log there?
brownts commentedon Apr 27, 2022
My suspicion is that this has to do with the timing changes that occurred between v0.25.1 and v0.26.0. In the previous version we'd try to start running 50ms after the debugger was started. This means, we weren't waiting for the rest of the initialization to complete (i.e., setting breakpoints, etc). My guess is that the
runcommand was executed earlier than it currently is.According to the normal "target remote mode" (Types of Remote Connections), the
runcommand is not supported. So, it likely shouldn't have worked before, but maybe the target server that this was used with allowed it, or allowed it before acontinuecommand. Anyway, my suspicion is that therunwas initiated earlier, possibly before thecontinuecommand, was therefore accepted...or at least didn't fail this way.Looking at the configuration, this really should be described as an
attach. I'd recommend starting with the following rewrite of the original configuration and see if that works as expected.abgironne commentedon May 3, 2022
I have the same problem using the esp32 and I can confirm that using type "attach" instead of type "launch" resolved it for me and everyone here.
GitMensch commentedon May 3, 2022
MAybe it would be good to inspect the launch configuration at start (we already do this) and when we get a launch inspect the autorun commands for an "attach" or "target" - and in this case create at least a stderr message that this likely should be adjusted to the attach configuration - or better do this with a popup and as the user if we should create a converted configuration...
If any of the windows users doing embedded debugging here could check if using gdb-multiarch from MSYS2 works at least as well a special debugger like xtensa-esp32-elf-gdb, this would be useful.
If it works, then you get the newest stable GDB (currently 12.1) with full python and python based source highlighting support, as well es defaulting to multithreading for reading symbol files which should speedup starting the debugger, too; and can also debug native Windows/Linux/other environments for both 32/64bit programs.
higaski commentedon Jul 4, 2022
I can confirm that changing the request to an "attach" also works for xtensa-esp32s3-elf-gdb.
higaski commentedon Jul 4, 2022
Well, or not. This is... weird.
I'm currently trying to use the built-in USB JTAG thing the new ESP32S3 chips have.
This works, but only if I start the OpenOCD server as SU. Native-Debug gives me a
Failed to get Stack Tracemessage and then I'm good to go.When I start OpenOCD as normal user the log looks exactly the same as when launching it as SU, but connecting GDB with Native-Debug suddenly gives me a
could not runmessage again and then the GDB connection gets dropped.GitMensch commentedon Jul 4, 2022
Your output says:
That's likely a good idea, see #330 for the documentation of how this needs to be setup (so far no one has provided a PR to get the missing pieces into README).
In any case - can you please provide the debug console log output from the extension in both cases so we see what is happening on the GDB side?
higaski commentedon Jul 4, 2022
Well... not really no. Don't get me wrong, I'd like to, I just discovered that changing the tiniest thing in the configuration leads to the GDB connection to get established. So, for example, setting
"showDevDebugOutput": trueor starting the OpenOCD server from a"preLaunchTask"makes everything work...Also, even if OpenOCD complains about the GDB connection getting dropped, simply running GDB again by re-running the launch configuration is enough to make it work.
Someone above mentioned something about a timeout or delay... this smells a lot like a timing issue to me.
GitMensch commentedon Jul 4, 2022
Hm, then I suggest to post the log with
"showDevDebugOutput": true(which would be expected to work) and one without (which would be expected to fail). Is the output on the gdb server side identical in both cases? Is there a "verbose" / "log" mode in this server to possibly see more?Any thoughts about using extended-remote?
higaski commentedon Jul 4, 2022
Oh I'm sorry. Now I get it. I thought the
showDevDebugOutputflag gives me the GDB output. Turns out the extension prints the GDB output to the Debug Console anyhow... I'm just stupid. 😄Not working. (started OpenOCD manually)
Working. (started OpenOCD with
preLaunchTask)Well, yes. Switching from this
to this
basically works. But again the latter seems to produce timings which result in the initial connection getting dropped...
brownts commentedon Jul 4, 2022
@higaski, looking at your logs above, it appears when you run OpenOCD manually, it starts running the inferior before the extension has even finished initialization (seeing that the "[New Thread ...]" lines show up before we've set the "app_main" breakpoint). I'm not familiar with OpenOCD or with your target, so just asking as part of trying to understand what normally happens on startup. I assume the target may start running, but that you issue "monitor reset halt" in the
autoruncommand list to get that under control. If that is the case, I'm surprised that some of the other commands continue to work (setting breakpoints) when the target is running, but maybe for that target, OpenOCD/GDB doesn't have a problem with that.Look at where the "[New Thread ...]" lines are between the two different logs. When you manually start, those occur much earlier in the log, which means the inferior has started running at that point in time. This happens before we can even set the breakpoint for "app_main" as part of the
stopAtEntryconfiguration. The extension has not told the debugger to run yet, so this must be happening by OpenOCD or something else in that path, not by the extension.In the second log, you can see the "[New Thread ...]" lines appearing after the breakpoint is set. That is consistent with what I would expect. As part of the
stopAtEntryconfiguration, we issue a breakpoint for "app_main" and then we send a "continue" to the debugger to run the inferior to the breakpoint. The second log looks normal as far as that is concerned.As far as the disconnect in the first log, I do have a suspicion about what might be happening. As part of the extension's default behavior (unless
stopAtConnectis used), the extension tries to run/continue execution. When I looked in the implementation, I don't believe we are necessarily waiting for theautoruncommands to complete before we issue the "continue". Therefore, I wonder if our issuing the "continue" may occur while still processing one of theautoruncommands.As an experiment to help prove if this is what is happening, can you make the following changes to your configuration and let me know if this fixes the issue:
autoruncommand."stopAtConnect": trueto your configuration.See if when you run this, and then manually continue (by pressing the continue button), if this fixes your problem. If so, this is likely an indication of my suspicion of the extension issuing the "continue" command before all of the
autoruncommands have finished being processed.higaski commentedon Jul 4, 2022
I've updated the launch config just as you've described and the behavior is exactly as predicted.
brownts commentedon Jul 9, 2022
@higaski, I've created an update that hopefully addresses the issue you are having. I'm hoping you can install the attached VSIX to confirm if this fixes your problem. If so, this will get rolled into the next official release. The current set of changes are in my forked branch here.
If you're unsure how to manually install an extension, refer to the Install from a VSIX. Note, you will need to rename the .zip extension to .vsix before following those instructions...apparently Github only allows attachments with certain file extensions and .vsix was not one of them.
debug-0.26.1-beta.0.zip
higaski commentedon Jul 10, 2022
Using the old configuration
this now works yes. The only time I can still trigger the "dropped connection" message is by trying to restart the target (green circle arrow button).
brownts commentedon Jul 10, 2022
I'm glad to hear that is working properly now. As for the "restart" scenario, do you have a log for that you can post? Does the "dropped connection" happen at the same spot as the previous scenario? I think the restart is equivalent to hitting the stop button and then relaunching the debug configuration...I assume that sequence works as expected.
GitMensch commentedon Jul 10, 2022
Actually there are two options depending on the support the debug extension claims. If it claims it supports a restart then an appropriate command is executed, if it doesn't then dap issues a stop and then a start command.
In both cases it could be that the server side has an issue, especially in the "supported" case if the extension does not execute the autorun commands and/or more are needed for the restart scenario.
brownts commentedon Jul 10, 2022
It would also be good to know if the "restart" behavior differs between the use of the "remote" configuration and the "extended remote" configuration. Looking at the internals for handling the disconnect request, it appears for an "extended remote" we send "-target-detach" and for the regular "remote", we just send "-gdb-exit". I wonder if the "dropped connection" only happens with one of these.
brownts commentedon Dec 14, 2022
I'm claiming this as fixed with f37ffb2, which is the proposed fix that was successfully verified to address the original issue. Feel free to create a new issue for the "restart" behavior if you are still seeing that problem.