feat(taskworker): Minimum Worker Concurrency

[george-sentry]

Linear

Refs STREAM-1269

Description

In #731, we deferred setting the status to SERVING for workers in push mode until all threads were warmed up because before, Kubernetes thought workers were ready to receive activations even though they weren't, causing significant throughput declines during worker redeployments.

Something similar happens when child processes recycle, or exit voluntarily because they have executed a certain, configurable number of tasks. This is meant to prevent memory leaks.

In some cases, child processes recycle at the same time, decreasing throughput significantly until all child processes are ready again. We can solve this problem by improving the child management process as follows.

• Spawn children and wait until they are all ready before advertising SERVING
• When a process has executed the maximum number of tasks specified, it does NOT exit right away - instead, it tells the parent that it's ready to exit and continues working
• The parent kills as many children as it can without falling below a configurable minimum concurrency (default is 0)

This way, after startup, there is some number of running child processes at all times. We may still see throughput dips during worker recycle waves, but the dip will be much less severe and somewhat predictable.

[linear-code]

STREAM-1269

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0f61d82. Configure here.}

[evanh]

Generally I think this is a good change (it all makes sense to me) but I think you could add more metrics. Some things I think would be useful:

• How long the child had to wait before the parent set the release event
• A gauge of how many children are in each state (pending/running/exiting)
• How many children are waiting to exit at any one time (exiting deque)