Add support for checking all living bthreads by ZhengweiZhu · Pull Request #3096 · apache/brpc

ZhengweiZhu · 2025-09-17T08:51:55Z

User can check all living bthreads by curl ip:port/bthreads/all or curl ip:port/bthreads/all?st=1 to show bthread stack trace. This is an enhancement of the original /bthreads service which provides a method to check a specified bthread by designated bthread id, as user has no idea what the bthread id is.

~~Condisering the performance cost brought by recording the bthread id on bthread startup and finish, currently this function is only enabled when BRPC_BTHREAD_TRACER is defined.~~

What problem does this PR solve?

brpc kindly provides a bthreads_service to check a specified thread by curl ip:port/bthreads/<bthread_id>. The problem is that we have no idea what the <bthread_id> is as it is generated by code, which makes this service useless.

Issue Number:
#3088

The sample output with stack trace (note that bthread in jumping status is not displayed due to implementation restriction, so it may not show all bthreads) :

# curl ip:port/bthreads/all?st=1
bthread=11716670785433 :
stop=0
interrupted=0
about_to_quit=0
fn=0x48ba90
arg=0x7f5b0401e380
attr={stack_type=3 flags=0 specified_tag=0 keytable_pool=0x3953570}
has_tls=0
uptime_ns=90442
cputime_ns=0
nswitch=0
status=5
traced=0
worker_tid=140029305263872
bthread call stack:
No frame
Error message: Forbid to trace self=11716670785433

bthread=4294969344 :
stop=0
interrupted=0
about_to_quit=0
fn=0x507850
arg=0
attr={stack_type=3 flags=0 specified_tag=-1 keytable_pool=0}
has_tls=0
uptime_ns=65724290725
cputime_ns=202946
nswitch=66
status=6
traced=0
worker_tid=140029437884160
bthread call stack:
# 0 0x0x5f6de1 bthread::TaskGroup::sched()
# 1 0x0x5f6f79 bthread::TaskGroup::usleep()
# 2 0x0x5ed514 bthread_usleep
# 3 0x0x508155 brpc::GlobalUpdate()
# 4 0x0x5f7989 bthread::TaskGroup::task_runner()
# 5 0x0x603401 bthread_make_fcontext

bthread=4294969345 :
stop=0
interrupted=0
about_to_quit=0
fn=0x501770
arg=0x3a22768
attr={stack_type=3 flags=320 specified_tag=0 keytable_pool=0}
has_tls=0
uptime_ns=65722683480
cputime_ns=65677492951
nswitch=13278
status=5
traced=0
worker_tid=140028751640320
bthread call stack:
# 0 0x0x7f5b26b7cc20 __restore_rt
# 1 0x0x2 <unknown>

bthread=4294969346 :
stop=0
interrupted=0
about_to_quit=0
fn=0x479dc0
arg=0x7ffec814c040
attr={stack_type=3 flags=0 specified_tag=0 keytable_pool=0}
has_tls=0
uptime_ns=65722871376
cputime_ns=7259150
nswitch=66
status=6
traced=0
worker_tid=140029305263872
bthread call stack:
# 0 0x0x5f6de1 bthread::TaskGroup::sched()
# 1 0x0x5f6f79 bthread::TaskGroup::usleep()
# 2 0x0x5ed514 bthread_usleep
# 3 0x0x47a84d brpc::Server::UpdateDerivedVars()
# 4 0x0x5f7989 bthread::TaskGroup::task_runner()
# 5 0x0x603401 bthread_make_fcontext

bthread=11557756996679 :
stop=0
interrupted=0
about_to_quit=0
fn=0x48ba90
arg=0x7f5b0401e080
attr={stack_type=3 flags=0 specified_tag=0 keytable_pool=0x3953570}
has_tls=0
uptime_ns=327293
cputime_ns=0
nswitch=0
status=5
traced=0
worker_tid=140029338834688
bthread call stack:
# 0 0x0x7f5b26b7cc20 __restore_rt
# 1 0x0x527685 brpc::InputMessenger::OnNewMessages()
# 2 0x0x48baa0 brpc::Socket::ProcessEvent()
# 3 0x0x5f7989 bthread::TaskGroup::task_runner()
# 4 0x0x603401 bthread_make_fcontext

Problem Summary:

What is changed and the side effects?

Changed:

add method butil::list_resources to list all object in the specified typed ResourcePool
filter out those bthreads created by bthread_start* functions
add /bthreads/all and /bthreads/all?st=1 service for user

Side effects:
No side effect now

Breaking backward compatibility: N/A

Check List:

Please make sure your changes are compilable.
When providing us with a new feature, it is best to add related tests.
Please follow Contributor Covenant Code of Conduct.

ZhengweiZhu · 2025-09-17T09:04:59Z

This pr will be useful to help diagnose bthread problem like bthread deadlock, which is hard or unrealistic by using gdb only.
However condisering the performance cost brought by lock, temporarily I just enable this feature when BRPC_BTHREAD_TRACER is defined. Do u guys have any suggestion to minimize the performance cost so let it be applicable for all situations. If so, I plan to add method to set bthread name and user can check all living bthreads with more detail, reducing the need to enable BRPC_BTHREAD_TRACER for stack trace.

wwbmmm · 2025-09-18T02:16:56Z

Maybe you can try to get all bthread_id from the butil::ResourcePool<bthread::TaskMeta>? (needs to add some interface to butil::ResourcePool)

chenBright · 2025-09-18T02:27:53Z

Maybe you can try to get all bthread_id from the butil::ResourcePool<bthread::TaskMeta>? (needs to add some interface to butil::ResourcePool)

I agree with the butil::ResourcePool<bthread::TaskMeta> approach.

yanglimingcn · 2025-09-18T11:00:13Z

yes, I think list all bthread_id is better。

ZhengweiZhu · 2025-09-18T14:25:21Z

Maybe you can try to get all bthread_id from the butil::ResourcePool<bthread::TaskMeta>? (needs to add some interface to butil::ResourcePool)

@wwbmmm @chenBright Yes I have thought of this approach but with no result.

Firstly, when we create a bthread, a ResourceId and TaskMeta instance is acquired from butil::ResourcePool by this way:
butil::ResourceId<TaskMeta> slot; TaskMeta* m = butil::get_resource(&slot);
If we know all the ResourceId slot then we can get all living bthread id by address_resource(slot)->tid. Inside the ResourcePool implementation, the slot is acquired by the following order: 1. thread local FreeChunk 2. global FreeChunk 3. thread local Block 4. global Block (from BlockGroup). The problem is the ResourcePool does not record the slot in use and it only knows the slot after it is not in use and returned to ResourcePool, which does not satisfy our requirement here.

Secondly, if we add some interface in ResourcePool to record the slot when get_resource and consider removing it when return_resource, certainly we might use thread local LocalPool to record for the sake of performance effect. The problem is the get_resource and return_resource may run in different pthread (different tls) as bthread switches to another worker. What's more, if we plan to get all living bthread id by summarizing all thread local records, the tricky part is the ResourcePool interface can be called from worker and non worker, making it even impossible to summarize.

Any ideas?

chenBright

I think you can traverse the global butil::ResourceId<TaskMeta>::_block_groups and get all tids.

ZhengweiZhu · 2025-09-22T16:23:53Z

I think you can traverse the global butil::ResourceId<TaskMeta>::_block_groups and get all tids.

That's huge. And they represent all bthread id which ever existed，not living one？

chenBright · 2025-09-23T02:14:12Z

I think you can traverse the global butil::ResourceId<TaskMeta>::_block_groups and get all tids.

That's huge. And they represent all bthread id which ever existed，not living one？

Maybe you can use TaskStatus to determine whether a bthread is alive.

chenBright · 2025-09-23T02:16:53Z

I think we can add a function to display all the live bthread ids and names, and click on the link to display the corresponding bthread details.

wwbmmm · 2025-09-23T03:02:38Z

I think you can traverse the global butil::ResourceId<TaskMeta>::_block_groups and get all tids.

That's huge. And they represent all bthread id which ever existed，not living one？

No. After a bthread exit, its TaskMeta will be return to the ResourcePool, and be reused by new bthread. When you traverse the ResourcePool, you only need to traverse those slots in use, you don't need to traverse those in the free list.

ZhengweiZhu · 2025-09-23T03:41:57Z

I think you can traverse the global butil::ResourceId<TaskMeta>::_block_groups and get all tids.

That's huge. And they represent all bthread id which ever existed，not living one？

No. After a bthread exit, its TaskMeta will be return to the ResourcePool, and be reused by new bthread. When you traverse the ResourcePool, you only need to traverse those slots in use, you don't need to traverse those in the free list.

How to judge if the slot is in use? @chenBright The TaskStatus is only used when TaskTracer is enabled, but I want this pr to be applicable even when TaskTracer is not enabled.

If I can judge if the slot is in use, then I need to traverse in the same way as describe_resources does?

ZhengweiZhu · 2025-09-23T03:49:18Z

I think we can add a function to display all the live bthread ids and names, and click on the link to display the corresponding bthread details.

If the TaskTracer is enabled, we can easily see call trace of all the living bthread and debug the deadlock problem, just like the same way as gdb/gstack does. And in normal situation, the living bthreads won't be many. Is it necessary then?

chenBright · 2025-09-25T02:25:12Z

I think we can add a function to display all the live bthread ids and names, and click on the link to display the corresponding bthread details.

If the TaskTracer is enabled, we can easily see call trace of all the living bthread and debug the deadlock problem, just like the same way as gdb/gstack does. And in normal situation, the living bthreads won't be many. Is it necessary then?

Yes. Because it allows users to view the call stack of a specified bthread, not all of them.

chenBright · 2025-09-25T02:26:55Z

The TaskStatus is only used when TaskTracer is enabled, but I want this pr to be applicable even when TaskTracer is not enabled.

Perhaps the default support for TaskStatus can meet your needs.

ZhengweiZhu · 2025-09-25T03:08:26Z

The TaskStatus is only used when TaskTracer is enabled, but I want this pr to be applicable even when TaskTracer is not enabled.

Perhaps the default support for TaskStatus can meet your needs.

Currently the TaskStatus is only set when TaskTracer is enabled. Do u mean I need to change the code to set some status when TaskTracer is not enabled? Like set status to TASK_STATUS_CREATED when create a bthread and set it status to TASK_STATUS_UNKNOWN when that bthread is destroyed, and a bthread can be judged alive when its status is not TASK_STATUS_UNKNOWN?

ZhengweiZhu · 2025-09-25T03:12:39Z

I think we can add a function to display all the live bthread ids and names, and click on the link to display the corresponding bthread details.

If the TaskTracer is enabled, we can easily see call trace of all the living bthread and debug the deadlock problem, just like the same way as gdb/gstack does. And in normal situation, the living bthreads won't be many. Is it necessary then?

Yes. Because it allows users to view the call stack of a specified bthread, not all of them.

I can add link to display the corresponding bthread details. And still display all living bthreads by default. The bthread name will not be shown and I can make another pr to add support naming bthread (and even execution_queue). Is that ok?

chenBright · 2025-09-25T09:57:53Z

The TaskStatus is only used when TaskTracer is enabled, but I want this pr to be applicable even when TaskTracer is not enabled.

Perhaps the default support for TaskStatus can meet your needs.

Currently the TaskStatus is only set when TaskTracer is enabled. Do u mean I need to change the code to set some status when TaskTracer is not enabled? Like set status to TASK_STATUS_CREATED when create a bthread and set it status to TASK_STATUS_UNKNOWN when that bthread is destroyed, and a bthread can be judged alive when its status is not TASK_STATUS_UNKNOWN?

Yes.

chenBright · 2025-09-25T09:58:32Z

I think we can add a function to display all the live bthread ids and names, and click on the link to display the corresponding bthread details.

If the TaskTracer is enabled, we can easily see call trace of all the living bthread and debug the deadlock problem, just like the same way as gdb/gstack does. And in normal situation, the living bthreads won't be many. Is it necessary then?

Yes. Because it allows users to view the call stack of a specified bthread, not all of them.

I can add link to display the corresponding bthread details. And still display all living bthreads by default. The bthread name will not be shown and I can make another pr to add support naming bthread (and even execution_queue). Is that ok?

No problem.

ZhengweiZhu · 2025-10-11T03:25:26Z

@wwbmmm @chenBright PTAL. The pr is updated and applicable no matter if BRPC_BTHREAD_TRACER is defined, with no performance side effect. Thanks for the suggestion. As for the idea to add link to display the corresponding bthread details, it's nice to have and probably be implemented later.

ZhengweiZhu · 2025-10-14T07:30:10Z

Maybe you can try to get all bthread_id from the butil::ResourcePool<bthread::TaskMeta>? (needs to add some interface to butil::ResourcePool)

There's a trick here. As each task group creates a TaskMeta object internally to run main task, these TaskMeta will also be traversed. I will do a filter to not show those TaskMeta as they are opaque to user.

ZhengweiZhu · 2025-10-16T05:28:53Z

The comments have all been resolved. But there's one problem left, that's when the bthread status is set to TASK_STATUS_READY, the actual status is set to TASK_STATUS_FIRST_READY.

brpc/src/bthread/task_tracer.cpp

Lines 150 to 154 in 0708333

    
           if (TASK_STATUS_READY == s || NULL == m->stack) { 
        
               m->status = TASK_STATUS_FIRST_READY; 
        
           } else { 
        
               m->status = s; 
        
           }

so the following judgement will never meet?

brpc/src/bthread/task_tracer.cpp

Lines 257 to 259 in 0708333

    
           } else if (TASK_STATUS_SUSPENDED == status || TASK_STATUS_READY == status) { 
        
               return ContextTrace(m->stack->context); 
        
           }

According to my test, there seems to always exist a bthread in TASK_STATUS_FIRST_READY status and not traceable. This bthread has flag 320 which means "BTHREAD_NEVER_QUIT | BTHREAD_GLOBAL_PRIORITY" , which seems to be EventDispatcher? Is this expected?

@chenBright

chenBright · 2025-10-16T08:15:29Z

The comments have all been resolved. But there's one problem left, that's when the bthread status is set to TASK_STATUS_READY, the actual status is set to TASK_STATUS_FIRST_READY.

brpc/src/bthread/task_tracer.cpp

Lines 150 to 154 in 0708333

if (TASK_STATUS_READY == s || NULL == m->stack) {

m->status = TASK_STATUS_FIRST_READY;

} else {

m->status = s;

}

so the following judgement will never meet?

brpc/src/bthread/task_tracer.cpp

Lines 257 to 259 in 0708333

} else if (TASK_STATUS_SUSPENDED == status || TASK_STATUS_READY == status) {

return ContextTrace(m->stack->context);

}

According to my test, there seems to always exist a bthread in TASK_STATUS_FIRST_READY status and not traceable. This bthread has flag 320 which means "BTHREAD_NEVER_QUIT | BTHREAD_GLOBAL_PRIORITY" , which seems to be EventDispatcher? Is this expected?

@chenBright

It should be if (TASK_STATUS_READY == s && NULL == m->stack), please fix it.

ZhengweiZhu · 2025-10-16T08:33:48Z

The comments have all been resolved. But there's one problem left, that's when the bthread status is set to TASK_STATUS_READY, the actual status is set to TASK_STATUS_FIRST_READY.

brpc/src/bthread/task_tracer.cpp

Lines 150 to 154 in 0708333

if (TASK_STATUS_READY == s || NULL == m->stack) {

m->status = TASK_STATUS_FIRST_READY;

} else {

m->status = s;

}

so the following judgement will never meet?

brpc/src/bthread/task_tracer.cpp

Lines 257 to 259 in 0708333

} else if (TASK_STATUS_SUSPENDED == status || TASK_STATUS_READY == status) {

return ContextTrace(m->stack->context);

}

According to my test, there seems to always exist a bthread in TASK_STATUS_FIRST_READY status and not traceable. This bthread has flag 320 which means "BTHREAD_NEVER_QUIT | BTHREAD_GLOBAL_PRIORITY" , which seems to be EventDispatcher? Is this expected?
@chenBright

It should be if (TASK_STATUS_READY == s && NULL == m->stack), please fix it.

Fixed! BTW fix another _enable_priority_queue not initialized bug. 😂

ZhengweiZhu · 2025-10-16T09:37:41Z

@wwbmmm @chenBright PTAL

wwbmmm · 2025-10-16T10:59:37Z

LGTM

User can check all living bthreads by `curl ip:port/bthreads/all` or when BRPC_BTHREAD_TRACER is enabled by `curl ip:port/bthreads/all?st=1` to show bthread stack trace. This is an enhancement of the original /bthreads service which provides a method to check a specified bthread by designated bthread id, as user has no idea what the bthread id is. BTW, fix _enable_priority_queue not initialized bug and fix task status incorrectly set to TASK_STATUS_FIRST_READY bug.

chenBright

LGTM

wwbmmm requested a review from chenBright September 18, 2025 02:17

chenBright reviewed Sep 22, 2025

View reviewed changes

ZhengweiZhu closed this Sep 25, 2025

ZhengweiZhu reopened this Sep 25, 2025

ZhengweiZhu mentioned this pull request Oct 9, 2025

bthread task tracer 存在死锁问题 #2983

Closed

ZhengweiZhu force-pushed the master branch from db7e37b to 64a12cb Compare October 10, 2025 10:27

wwbmmm reviewed Oct 12, 2025

View reviewed changes

chenBright reviewed Oct 12, 2025

View reviewed changes

Comment thread src/bthread/task_control.h Outdated

Comment thread src/bthread/task_group.cpp

ZhengweiZhu force-pushed the master branch 2 times, most recently from 4106c6f to 252a2d2 Compare October 14, 2025 07:58

chenBright reviewed Oct 14, 2025

View reviewed changes

Comment thread src/bthread/task_control.cpp Outdated

Comment thread src/bthread/task_group.cpp Outdated

Comment thread src/bthread/bthread.cpp Outdated

Comment thread src/bthread/task_control.cpp Outdated

ZhengweiZhu force-pushed the master branch from 252a2d2 to 5e91f86 Compare October 16, 2025 03:58

wwbmmm reviewed Oct 16, 2025

View reviewed changes

Comment thread src/bthread/task_control.cpp Outdated

ZhengweiZhu force-pushed the master branch from 5e91f86 to 0f1c3e5 Compare October 16, 2025 08:32

ZhengweiZhu mentioned this pull request Oct 16, 2025

Add flag for bthread priority queue, the default is false #3078

Merged

ZhengweiZhu force-pushed the master branch from 0f1c3e5 to 0c61ca4 Compare October 16, 2025 09:28

chenBright reviewed Oct 16, 2025

View reviewed changes

Comment thread src/bthread/task_control.cpp Outdated

ZhengweiZhu force-pushed the master branch from 0c61ca4 to e4634df Compare October 17, 2025 01:45

ZhengweiZhu force-pushed the master branch from e4634df to 94718fe Compare October 17, 2025 05:30

ZhengweiZhu requested a review from chenBright October 19, 2025 01:25

chenBright approved these changes Oct 21, 2025

View reviewed changes

chenBright merged commit 5f1d893 into apache:master Oct 22, 2025
16 checks passed

yannan-wyn mentioned this pull request Apr 15, 2026

fix(bthread): refactor sharded priority queue with lock-free MPSC inb… #3270

Open

Conversation

ZhengweiZhu commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and the side effects?

Check List:

Uh oh!

ZhengweiZhu commented Sep 17, 2025

Uh oh!

wwbmmm commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chenBright commented Sep 18, 2025

Uh oh!

yanglimingcn commented Sep 18, 2025

Uh oh!

ZhengweiZhu commented Sep 18, 2025

Uh oh!

chenBright left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ZhengweiZhu commented Sep 22, 2025

Uh oh!

chenBright commented Sep 23, 2025

Uh oh!

chenBright commented Sep 23, 2025

Uh oh!

wwbmmm commented Sep 23, 2025

Uh oh!

ZhengweiZhu commented Sep 23, 2025

Uh oh!

ZhengweiZhu commented Sep 23, 2025

Uh oh!

chenBright commented Sep 25, 2025

Uh oh!

chenBright commented Sep 25, 2025

Uh oh!

ZhengweiZhu commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ZhengweiZhu commented Sep 25, 2025

Uh oh!

chenBright commented Sep 25, 2025

Uh oh!

chenBright commented Sep 25, 2025

Uh oh!

ZhengweiZhu commented Oct 11, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZhengweiZhu commented Oct 14, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ZhengweiZhu commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

chenBright commented Oct 16, 2025

Uh oh!

ZhengweiZhu commented Oct 16, 2025

Uh oh!

ZhengweiZhu commented Oct 16, 2025

Uh oh!

wwbmmm commented Oct 16, 2025

Uh oh!

Uh oh!

chenBright left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ZhengweiZhu commented Sep 17, 2025 •

edited

Loading

wwbmmm commented Sep 18, 2025 •

edited

Loading

chenBright left a comment •

edited

Loading

ZhengweiZhu commented Sep 25, 2025 •

edited

Loading

ZhengweiZhu commented Oct 16, 2025 •

edited

Loading