Support Lambda Managed Instances in DurableExecution by normj · Pull Request #2605 · aws/aws-lambda-dotnet

normj · 2026-11-03T02:21:01Z

Summary

Makes Amazon.Lambda.DurableExecution safe to run on Lambda Managed Instances (LMI), where several invocations share one process and Lambda does not stop an invocation that times out. Addresses the source-review findings in #2604.

1. Cancellation lost at teardown (all modes)

TerminationManager completes its task with RunContinuationsAsynchronously, so WorkflowCancellation received the cancel through a threadpool continuation. If the wrapper disposed the cancellation source first, the continuation hit ObjectDisposedException, which was swallowed, and the cancel was lost. The wrapper now cancels synchronously (BeginTeardown) before draining and disposing, and Dispose also cancels before releasing the source.

2. Abandoned work after the invocation ends (all modes)

When a workflow suspends, user work it started but did not await (a step still running, a losing parallel branch) kept running. On LMI the process stays live, so that work runs alongside other invocations.

The workflow token is cancelled when the invocation ends.
The token is captured at construction, so stale code that reads it after disposal gets a cancelled token instead of ObjectDisposedException.
Every IDurableContext operation entry point rejects new operations after teardown with OperationCanceledException.
The abandoned workflow task's fault is observed, so it can't surface through UnobservedTaskException in another invocation.

3. Invocation deadline (LMI only)

Only when AWS_LAMBDA_MAX_CONCURRENCY is set; on-demand behavior is unchanged.

The deadline comes from ILambdaContext.RemainingTime. Zero or negative means no deadline, so an unconfigured TestLambdaContext doesn't cancel immediately.
At the deadline: the workflow token is cancelled, new operations are rejected, and state hydration (GetExecutionStateAsync) is cancelled.
Checkpoint drain is bounded by the time left until the deadline plus a 2 second cleanup budget. Past that, the flush is cancelled and the batcher worker is abandoned rather than awaited.
WrapAsync throws TimeoutException. This escapes as an invocation error, the same outcome as an on-demand timeout, and is never recorded as a Failed durable execution, even if user code catches the cancellation and throws its own exception.
User code gets the full deadline; there's no early-firing margin.

Testing

New LambdaManagedInstancesTests (23 tests) drive DurableFunction concurrently with an in-memory service client:
- Concurrent invocations through one shared handler, with every invocation inside a step at the same time: execution ARN, request ID, operation IDs, payloads and checkpoint token chains stay isolated.
- A peer timing out while a healthy peer keeps running and succeeds.
- Replay of one execution while a fresh peer is live: completed steps are skipped.
- Teardown ordering (cancel before dispose, token usable after dispose, new operations rejected) in both modes.
- Deadline during a step, during hydration, a stalled checkpoint call that ignores cancellation, and user code converting the cancellation into its own exception.
- On-demand: no deadline, and hydration gets a non-cancellable token.
LMI mode is selected through an internal options seam rather than the process-wide environment variable, so the tests can run in parallel. Ordering uses signals, not sleeps, and no test asserts on elapsed time.
I confirmed the guards catch the bugs by reverting each fix: the race, token-after-dispose, admission and deadline-fault-mapping tests fail without their fixes.
Amazon.Lambda.DurableExecution.Tests (524) and Amazon.Lambda.DurableExecution.Testing.Tests (154) pass on net8.0 and net10.0. The new tests passed 10 repeated runs.

Not covered

No test that provisions a real LMI environment, by design: it would take too long in CI.
Code that ignores the token or blocks synchronously still can't be stopped; this is documented in docs/core/cancellation.md.
IDurableParallel branches added to an existing incremental parallel after teardown aren't gated at entry; their child operations are.
I didn't run the Native AOT publish test. The change adds no reflection.

- Cancel the workflow token synchronously at teardown, before disposal. Termination delivered the cancel via a threadpool continuation that could run after disposal and be silently dropped. - Clean up abandoned work: keep the workflow token readable after dispose, reject new durable operations once the invocation ends, and observe the abandoned workflow task's fault. - Lambda Managed Instances only: apply the invocation deadline from RemainingTime. Cancel workflow work and state hydration at the deadline, bound checkpoint cleanup, and throw TimeoutException so the invocation is retried rather than recorded as a durable failure. On-demand behavior is unchanged. - Add multi-concurrency unit tests for isolation, peer timeouts, replay alongside a live peer, and teardown ordering. Refs #2604

Copilot

🟡 Changes recommended

Critical deadline and cancellation races can leave invocations running indefinitely, produce incorrect outcomes, and allow work or checkpoint writes after teardown.

12 open issues

What changed in this PR

Adds deadline and teardown handling to Amazon.Lambda.DurableExecution for Lambda Managed Instances, addressing part of #2604’s shared-process execution concerns.

Changes:

Adds invocation-scoped deadlines, timeout reporting, and bounded checkpoint cleanup.
Cancels abandoned workflow work and rejects new durable operations after teardown.
Adds concurrency and lifecycle tests, cancellation guidance, and patch release notes.

File	Description
`Libraries/test/Amazon.Lambda.DurableExecution.Tests/LambdaManagedInstancesTests.cs`	Tests invocation isolation, replay, deadlines, and teardown.
`Libraries/src/Amazon.Lambda.DurableExecution/Internal/WorkflowCancellation.cs`	Tracks cancellation, teardown, and LMI deadlines.
`Libraries/src/Amazon.Lambda.DurableExecution/Internal/DurableInvocationOptions.cs`	Provides test overrides for LMI detection and cleanup budgets.
`Libraries/src/Amazon.Lambda.DurableExecution/Internal/CheckpointBatcher.cs`	Adds timed draining and worker abandonment.
`Libraries/src/Amazon.Lambda.DurableExecution/DurableFunction.cs`	Integrates deadlines, hydration cancellation, and cleanup.
`Libraries/src/Amazon.Lambda.DurableExecution/DurableExecutionHandler.cs`	Races workflow execution against deadlines and observes abandoned faults.
`Libraries/src/Amazon.Lambda.DurableExecution/DurableContext.cs`	Guards operation entry points against teardown.
`Libraries/src/Amazon.Lambda.DurableExecution/docs/core/cancellation.md`	Documents LMI cancellation behavior and cooperative-code limits.
`.autover/changes/46bb3a38-e1f3-48d1-ab9b-10a36475e901.json`	Records the patch release changes.

_{Review effort: Balanced}

- Contain exceptions thrown by user cancellation callbacks so they cannot crash the shared process from the deadline timer or replace the outcome during teardown. - Signal the deadline before cancelling, so a blocking user callback cannot stop the handler from seeing it; unregister rather than dispose the deadline registration. - Read IsDeadlineExceeded from the deadline token, independent of which cancellation reason came first. - Let the deadline decide the outcome whichever task wins the race. - Treat an already-expired LMI invocation as an immediate timeout. - Do not checkpoint FAIL or RETRY for exceptions raised after the deadline in steps, child contexts and condition checks. - Re-check cancellation before running a step body, child function or condition check, and when the queued workflow actually starts. - Re-check the deadline after the final drain, bound hydration even when the client ignores cancellation, and stop the batcher from sending further batches once it is shut down. - Tests: add a regression test per fix and a hang guard that does not throw TimeoutException, so a hang can no longer pass a deadline test.

Copilot

🟡 Changes recommended

Cancellation callbacks can indefinitely block teardown or terminate the process, while deadline races can persist incorrect failure or retry checkpoints.

12 open issues

11 resolved since last review

_{Review effort: Balanced}

- Treat an OCE as cancellation when the workflow token is cancelled even if the linked token has not been notified yet (step, child context, condition check), so a body that never ran is never recorded as failed. - Map/parallel: check the workflow token directly before recording the parent outcome, matching IncrementalParallelOperation. - Condition checks: do not record RETRY/FAIL/SUCCEED for a check that returned after the LMI deadline. - Own the deadline timer so a throwing deadline-token callback (for example from the service client) cannot crash the process. - Checkpoint batcher: cancel an abandoned flush with CancelAsync so blocking client callbacks cannot stall the drain, and release pending EnqueueAsync awaiters on abandonment. - Use one combined deadline + cleanup budget for the final drain. - Report a timeout when output serialization crosses the deadline. - Always observe an abandoned hydration request's fault. - Tests: regression test per fix; the deadline fault-race test registers on the deadline token so it exercises the race again; add a teardown-path test for throwing cancellation callbacks.

Copilot

🟡 Changes recommended

Blocking cancellation callbacks can hang teardown and deadline handling, while deadline races can allow expired workflows to start, succeed, or persist outcomes.

7 open issues

11 resolved since last review

_{Review effort: Balanced}

Third round of Copilot review findings, addressed with one rule instead of per-site patches: the SDK never runs cancellation callbacks it does not own (user code, the service client) on its own threads, and it publishes its own deadline and teardown state before any foreign code runs. - Every cancel the SDK starts uses CancellationTokenSource.CancelAsync: tokens report cancellation immediately, callbacks run on the threadpool, and their exceptions are logged. BeginTeardown can no longer block in a user callback, in any mode. - A short lock around starting the cancel ensures every caller returns only after the token reports cancellation. - Sources are disposed only after callback delivery finishes, so no callback is skipped. - The deadline is anchored to a monotonic timestamp taken when RemainingTime was read. IsDeadlineExceeded and admission compare the clock, and fire the deadline themselves if the timer is late. - Firing the deadline records the reason, closes admission and signals DeadlineTask before cancelling the workflow token, then the client's deadline token. Hydration races DeadlineTask, not the client token. - Map/parallel parents check the deadline before recording an outcome. - Condition checks re-check the deadline after serializing state, on the RETRY, SUCCEED and max-attempts FAIL paths. - Checkpoint batcher always observes an abandoned item's fault. - Tests: deadline-dependent tests fire the deadline themselves after setup (TriggerDeadline) instead of racing a real 50 ms timer, and a test-only option leaves the timer unarmed to model a late timer.

Copilot

🟡 Changes recommended

Critical deadline-handling races can leave hydration uncancelled and allow step, child, and condition callbacks to start after expiration.

6 open issues

6 resolved since last review

_{Review effort: Balanced}

- Start the deadline-token and workflow-token cancels before signalling DeadlineTask, so a waiter that ends the invocation and disposes at once always finds the deadline cancel under way. Dispose and starting a cancel now coordinate under one lock: a cancel is either already started (and waited for) or never starts on a released source. - Check the clock-based deadline immediately before running a step body, a child context function or a condition check, alongside the token check, so an awaited START or a replay state restore that crosses the deadline before its timer runs cannot start user code. - Tests: barrier test disposing right after DeadlineTask is signalled; unarmed-timer tests for step and child START waits and condition replay; the step-deadline wrapper test now fires a controllable deadline after the step is entered instead of racing a real 100 ms deadline.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot

🟡 Changes recommended

A moderate issue remains in deadline handling because hydration faults after the deadline can escape as SDK errors instead of being consistently translated into the promised timeout exception.

1 open issue

Post-deadline hydration faults escape timeout translation · New

_{Review effort: Lite}

+        catch (OperationCanceledException ex) when (workflowCancellation.IsDeadlineExceeded)
        {
-            var (operations, marker) = await serviceClient.GetExecutionStateAsync(
-                invocationInput.DurableExecutionArn, checkpointToken, nextMarker);
-            state.AddOperations(operations);
-            nextMarker = marker;
+            throw CreateDeadlineException(invocationInput.DurableExecutionArn, ex);


normj requested a balanced review from Copilot November 3, 2026 23:35

Copilot started reviewing on behalf of normj November 3, 2026 23:37

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

normj requested a balanced review from Copilot November 4, 2026 01:02

Copilot started reviewing on behalf of normj November 4, 2026 01:03

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

normj requested a balanced review from Copilot November 4, 2026 02:02

Copilot started reviewing on behalf of normj November 4, 2026 02:02

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

normj requested a balanced review from Copilot November 4, 2026 18:41

Copilot started reviewing on behalf of normj November 4, 2026 18:42

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

normj requested a balanced review from Copilot November 5, 2026 00:18

Copilot AI reviewed Nov 5, 2026

normj requested a balanced review from Copilot November 5, 2026 00:33

Copilot AI reviewed Nov 5, 2026

normj requested a balanced review from Copilot November 5, 2026 02:25

Copilot AI reviewed Nov 5, 2026

normj requested a lite review from Copilot November 5, 2026 02:26

Copilot started reviewing on behalf of normj November 5, 2026 02:26

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

Conversation

normj commented Nov 3, 2026

Summary

1. Cancellation lost at teardown (all modes)

2. Abandoned work after the invocation ends (all modes)

3. Invocation deadline (LMI only)

Testing

Not covered

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants