fix: await worker startup readiness - #353
Conversation
There was a problem hiding this comment.
Pull request overview
This PR changes TaskHubGrpcWorker.start() semantics so that awaiting startup only resolves after the sidecar “hello” handshake succeeds and the initial work-item stream is confirmed ready, with a configurable startup timeout budget (default 30s). It also wires the new timeout knob through the Azure-managed worker builder and updates docs/changelogs accordingly.
Changes:
- Make
await worker.start()wait for hello + initial work-item stream readiness, with a boundedstartupTimeoutMscovering metadata generation, hello, and stream readiness. - Add comprehensive unit tests for startup timing, cancellation, late callbacks, and stream recovery behavior.
- Expose
.startupTimeout(milliseconds)onDurableTaskAzureManagedWorkerBuilderand document the new startup semantics.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Documents new start() readiness semantics and builder timeout configuration. |
| CHANGELOG.md | Notes the core worker startup behavior change. |
| packages/durabletask-js/src/worker/task-hub-grpc-worker.ts | Implements startup readiness gating, startup timeout budget, and generation-based cancellation/cleanup. |
| packages/durabletask-js/test/worker-startup.spec.ts | Adds focused tests covering startup readiness, timeouts, stop-during-start, and late callbacks. |
| packages/durabletask-js/test/worker-stream-recovery.spec.ts | Adjusts mocks to account for metadata-based “stream ready” detection. |
| packages/durabletask-js-azuremanaged/src/worker-builder.ts | Adds .startupTimeout() builder method and passes it into core worker options. |
| packages/durabletask-js-azuremanaged/test/unit/worker-builder.spec.ts | Verifies builder propagates startup timeout to the core worker instance. |
| packages/durabletask-js-azuremanaged/README.md | Documents startup readiness and builder timeout configuration. |
| packages/durabletask-js-azuremanaged/CHANGELOG.md | Notes the new builder API for startup timeout configuration. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 10 changed files in this pull request and generated no new comments.
Suppressed comments (2)
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts:425
- The start()
@throwsdoc is too narrow: start() can now throw TimeoutError for metadata generation and work-item stream establishment timeouts as well, not only the hello handshake.
* @throws {@link TimeoutError} if the hello handshake exceeds `startupTimeoutMs`.
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts:71
- The TaskHubGrpcWorkerOptions.startupTimeoutMs doc comment no longer matches behavior: the timeout budget is also applied to work-item stream metadata generation and establishing the initial stream, not just hello/metadata. Updating the comment helps avoid confusing API consumers.
/**
* Optional timeout in milliseconds for initial metadata generation and the sidecar hello handshake.
* Defaults to 30000.
*/
startupTimeoutMs?: number;
Make worker startup await the initial sidecar stream, bound connection setup with a configurable timeout, and cleanly abort failed or stopped attempts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Treat an idle stream as ready once hello succeeds and handlers are attached, preserve retryability after client construction failures, and align the recovery E2E with the startup contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
Add an isolated job that leaves the emulator stopped initially so the worker startup failure and subsequent stream recovery path execute in CI. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
Document the startup rejection as a breaking change, align timeout API wording, and preserve the orchestration reuse documentation while rebasing onto main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 65b88a3f-d79a-4a63-8636-cc41c219db63
bbef765 to
3482ecb
Compare
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
45003c4 to
82e60fd
Compare
Track direct internal worker runs under the same lifecycle invariants as start and reconnect so stop can drain them and later starts cannot race stale streams. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: d491ec0d-9ce9-421b-9953-7d179d53625b
| run: npx jest ${{ matrix.test-group.pattern }} --runInBand --detectOpenHandles | ||
| timeout-minutes: 15 | ||
|
|
||
| worker-stream-recovery: |
There was a problem hiding this comment.
why this tests is separate
Summary
await worker.start()resolve after startup metadata is generated, the hello handshake succeeds, and the initial work-item stream is created with all handlers attachedstartupTimeoutMsbudget for metadata generation and hello, with gRPC deadline/cancellationstartupTimeout()through the Azure-managed worker builder and document the new semanticsBreaking change
Previously,
await worker.start()resolved before the sidecar was connected and retried forever in the background. It now rejects when the sidecar cannot be reached withinstartupTimeoutMs(30 seconds by default). Deployments where the worker can start before its sidecar should increase the timeout or retry startup from the caller:Fixes #69
Validation
npm test -w @microsoft/durabletask-js -- --runTestsByPath test/worker-startup.spec.ts test/worker-stream-recovery.spec.ts test/worker-unhandled-rejection.spec.ts— 3 suites / 26 tests passednpm test -w @microsoft/durabletask-js-azuremanaged -- --runTestsByPath test/unit/worker-builder.spec.ts— 1 suite / 7 tests passednpm run build:core && npm run build:azuremanagedgit diff --check