Breakpoints are randomly not set by the extension

Edit: Could be related to #276 (comment) and #268

Although a breakpoint is set in Visual Studio Code before the debugging task is executed, only sometimes the extension will instruct gdb to insert a breakpoint.

Running VSCode Version 1.55.2, gdb (Ubuntu 10.2-0ubuntu1~ 20.04 ~ 1) and extension version 0.25.1 from the VSCode extension marketplace.

If you are using gdb
- gdb --version >= 7.7.1
- it works on the command line with gdb
- cwd and target are properly set
If you are using lldb
- lldb --version >= 3.7.1
- it works on the command line with lldb-mi
- cwd and target are properly set

Configuration:

            "name": "x86",
            "request": "launch",
            "target": "${command:cmake.launchTargetPath}",
            "type": "gdb",
            "printCalls": true,
            "showDevDebugOutput": true,

and "debug.allowBreakpointsEverywhere": true,

Output:

1-gdb-set target-async on
2-environment-directory "/workspace"
3-file-exec-and-symbols "/workspace/build/executable"
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-group-added","output":[["id","i1"]]}]}
GDB -> App: {"token":1,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[]}}
GDB -> App: {"token":2,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[["source-path","/workspace:$cdir:$cwd"]]}}
GDB -> App: {"token":3,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[]}}
Running executable
4-exec-run
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-group-started","output":[["id","i1"],["pid","24133"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-created","output":[["id","1"],["group-id","i1"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib64/ld-linux-x86-64.so.2"],["target-name","/lib64/ld-linux-x86-64.so.2"],["host-name","/lib64/ld-linux-x86-64.so.2"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7fd0100"],["to","0x00007ffff7ff2674"]]]]]}]}
GDB -> App: {"token":4,"outOfBandRecord":[],"resultRecords":{"resultClass":"running","results":[]}}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"exec","asyncClass":"running","output":[["thread-id","all"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libpthread.so.0"],["target-name","/lib/x86_64-linux-gnu/libpthread.so.0"],["host-name","/lib/x86_64-linux-gnu/libpthread.so.0"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7fa5ae0"],["to","0x00007ffff7fb54d5"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libstdc++.so.6"],["target-name","/lib/x86_64-linux-gnu/libstdc++.so.6"],["host-name","/lib/x86_64-linux-gnu/libstdc++.so.6"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7e5a160"],["to","0x00007ffff7f42452"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libm.so.6"],["target-name","/lib/x86_64-linux-gnu/libm.so.6"],["host-name","/lib/x86_64-linux-gnu/libm.so.6"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7c7c3c0"],["to","0x00007ffff7d22f18"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libgcc_s.so.1"],["target-name","/lib/x86_64-linux-gnu/libgcc_s.so.1"],["host-name","/lib/x86_64-linux-gnu/libgcc_s.so.1"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7c555e0"],["to","0x00007ffff7c66045"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libc.so.6"],["target-name","/lib/x86_64-linux-gnu/libc.so.6"],["host-name","/lib/x86_64-linux-gnu/libc.so.6"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7a85630"],["to","0x00007ffff7bfa20d"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":true,"type":"console","content":"[Thread debugging using libthread_db enabled]\n"}]}
[Thread debugging using libthread_db enabled]
GDB -> App: {"outOfBandRecord":[{"isStream":true,"type":"console","content":"Using host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\n"}]}
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
GDB -> App: {"outOfBandRecord":[{"isStream":true,"type":"console","content":"[Inferior 1 (process 24133) exited normally]\n"}]}
[Inferior 1 (process 24133) exited normally]
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-exited","output":[["id","1"],["group-id","i1"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-group-exited","output":[["id","i1"],["exit-code","0"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"exec","asyncClass":"stopped","output":[["reason","exited-normally"]]}]}
-gdb-exit
GDB -> App: {"outOfBandRecord":[],"resultRecords":{"resultClass":"exit","results":[]}}
5-thread-info

Rerunning it without changing anything sometimes leads to a successful breakpoint insertion:

1-gdb-set target-async on
2-environment-directory "/workspace"
3-file-exec-and-symbols "/workspace/build/executable"
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-group-added","output":[["id","i1"]]}]}
GDB -> App: {"token":1,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[]}}
GDB -> App: {"token":2,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[["source-path","/workspace:$cdir:$cwd"]]}}
GDB -> App: {"token":3,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[]}}
4-break-delete
GDB -> App: {"token":4,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[]}}
5-break-insert -f "/workspace/Tests/RunAllTests.cpp:30"
GDB -> App: {"token":5,"outOfBandRecord":[],"resultRecords":{"resultClass":"done","results":[["bkpt",[["number","1"],["type","breakpoint"],["disp","keep"],["enabled","y"],["addr","0x000000000003a667"],["func","main(int, char**)"],["file","../../Tests/RunAllTests.cpp"],["fullname","/workspace/Tests/RunAllTests.cpp"],["line","30"],["thread-groups",["i1"]],["times","0"],["original-location","/workspace/Tests/RunAllTests.cpp:30"]]]]}}
Running executable
6-exec-run
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-group-started","output":[["id","i1"],["pid","25360"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"thread-created","output":[["id","1"],["group-id","i1"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"breakpoint-modified","output":[["bkpt",[["number","1"],["type","breakpoint"],["disp","keep"],["enabled","y"],["addr","0x000055555558e667"],["func","main(int, char**)"],["file","../../Tests/RunAllTests.cpp"],["fullname","/workspace/Tests/RunAllTests.cpp"],["line","30"],["thread-groups",["i1"]],["times","0"],["original-location","/workspace/Tests/RunAllTests.cpp:30"]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib64/ld-linux-x86-64.so.2"],["target-name","/lib64/ld-linux-x86-64.so.2"],["host-name","/lib64/ld-linux-x86-64.so.2"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7fd0100"],["to","0x00007ffff7ff2674"]]]]]}]}
GDB -> App: {"token":6,"outOfBandRecord":[],"resultRecords":{"resultClass":"running","results":[]}}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"exec","asyncClass":"running","output":[["thread-id","all"]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libpthread.so.0"],["target-name","/lib/x86_64-linux-gnu/libpthread.so.0"],["host-name","/lib/x86_64-linux-gnu/libpthread.so.0"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7fa5ae0"],["to","0x00007ffff7fb54d5"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libstdc++.so.6"],["target-name","/lib/x86_64-linux-gnu/libstdc++.so.6"],["host-name","/lib/x86_64-linux-gnu/libstdc++.so.6"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7e5a160"],["to","0x00007ffff7f42452"]]]]]}]}
GDB -> App: {"outOfBandRecord":[{"isStream":false,"type":"notify","asyncClass":"library-loaded","output":[["id","/lib/x86_64-linux-gnu/libm.so.6"],["target-name","/lib/x86_64-linux-gnu/libm.so.6"],["host-name","/lib/x86_64-linux-gnu/libm.so.6"],["symbols-loaded","0"],["thread-group","i1"],["ranges",[[["from","0x00007ffff7c7c3c0"],["to","0x00007ffff7d22f18"]]]]]}]}

Adding "showDevDebugOutput": true, increased the probability that the breakpoints were inserted -> Race Condition?!?

Also tested shortly with the latest master (15f7bae), but ran sometimes into issues that the debugging session never started.

@brownts

the debugReady seems to be set once the debug-ready event is emitted (happens after the load commands are ran in GDB, like loading the symbols, loading path translations, preparing async exec, etc. The if simply checks, if the event has been fired already, in which case to call the code immediately, otherwise wait for the event and call it once the event triggers once.

Running immediately instead of waiting for the event could mean that some configuration is skipped and with that bugs with the break insert functions could arise for example when the path translations aren't sent yet to GDB. I think your solution here is working for you because you are giving the callback more time in the race to finish, but it's not a solid solution and it will break once the break command is dependent on some of the previous setup commands. (like with SSH, WSL or MSYS/MinGW setups)

I would have thought this should be fixed on master with #304 being merged, after reading it might actually be that that is the cause for the race being lost by the break function now? (cc @brownts)

~~If that PR started this issue for you though~~, (PR can't be the reason it started happening because it's not in the currently released version where you said it also happens) reverting would not be a good choice - rather the underlying race condition should be solved. (wait for running for breaks, etc) It looks like before there was a 50ms timeout that may have been used to make this race work, which is not clean and error prone I think.

I took a look at where the "debug-ready" event is emitted and in this case it's the load routine. I suspect there might be a race condition between when that event is emitted and when the breakpoint requests sent from VSCode are processed. It appears the only way that debugReady can be set to "true" is via one of these breakpoint callbacks. Thus, I believe in order for this to work properly, the breakpoint request must be received prior to the "debug-ready" event emitted from the load routine (so the callback is registered with the event prior to the event firing). Otherwise, the event will have fired in the past, but the debugReady boolean will never have been set true (since no callback had been registered with that event yet). Then when the breakpoint request(s) are processed the boolean is still false and the callback is registered to fire after the "debug-ready" event occurs (but it never occurs again since it already happened).

That's my suspicion about what is going on here. It seems to me that the setting of the debugReady boolean should happen independently of the breakpoint callback, when the "debug-ready" event fires, to avoid this potential race condition.

A simple test to prove out this theory would be to add a handler for the "debug-ready" event that just sets this flag. Something like the following (untested of course 😉) added to the mibase.ts:initDebugger):

this.miDebugger.on("debug-ready", (() => { this.debugReady = true; }));

@WebFreak001

Hi @JanHildenbrandBosch, no problem on the response time, I'm just glad we understand the problem and were able to get you unstuck.

@WebFreak001, I'm wondering why we implement the breakpoint setting in this manner (by potentially registering callbacks with the event system to handle them at a later time). It seems if we just delayed the sending of the "initialized" event back to the editor, we wouldn't have to add callbacks based on the debugReady boolean.

According to the DAP Specification section on "Configuring breakpoint and exception behavior", it indicates:

Since the development tool does not know when is the correct moment for passing the configuration information to the adapter, the debug adapter is expected to send an initialized event to the development tool to announce that it is ready to accept configuration requests. With this approach a debug adapter does not have to implement a buffering strategy for configuration information.

Per the wording above, the protocol is supposed to be flexible enough to avoid having to "... implement a buffering strategy ...", which seems like what this is doing by adding the callbacks to the event. Currently, we send the "initialized" event during the early part of the "launch" or "attach", but I wonder if we should instead send it when the "debug-ready" event fires. This should guarantee that no breakpoint configuration is sent by the editor until after the debugger has been completely initialized. If we did that, I don't think we'd even need the debugReady boolean nor require callbacks for the setting of the breakpoints.

Maybe sending this event early was necessary when the 50ms timer was in place, however with the current implementation, the debugger will not be commanded to start running until after the configuration has completed.

@brownts

Just a note: I think it is time for a new release once this issue is set. @brownts would you consider a PR?

@brownts

Just a note: I think it is time for a new release once this issue is set. @brownts would you consider a PR?

@GitMensch, can we resolve PR #259 before we create a PR for this issue, so we aren't having to resolve merge conflicts? There is some overlap between that PR and what would be changed here. It would be good to incorporate that PR for the next release as well.

@brownts

@brownts Sure, the related PR is merged now :-)

Navigation Menu

Repository navigation

Breakpoints are randomly not set by the extension #307

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Breakpoints are randomly not set by the extension #307

Description

Activity

WebFreak001 commented on Dec 23, 2021

brownts commented on Dec 23, 2021

brownts commented on Jan 3, 2022

GitMensch commented on Jan 4, 2022

brownts commented on Jan 4, 2022

GitMensch commented on Feb 25, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions