Skip to content

fix(daemon): keep runtimeless drops retiring until the drain confirms - #651

Merged
ScriptedAlchemy merged 1 commit into
codex/tracedecay-total-redesign-planfrom
codex/observability-runtimeless-drop
Aug 22, 2026
Merged

fix(daemon): keep runtimeless drops retiring until the drain confirms#651
ScriptedAlchemy merged 1 commit into
codex/tracedecay-total-redesign-planfrom
codex/observability-runtimeless-drop

Conversation

@ScriptedAlchemy

Copy link
Copy Markdown
Owner

Summary

Resolves the unresolved Codex P2 finding on #650: "Keep runtimeless drops retiring until the owner closes."

The runtimeless-drop path shipped in #650 called finish_retirement(&core, true) when the last alias handle was dropped outside a tokio runtime — treating inability-to-await as a successful retirement. That vacated the exact-store entry while a retained producer frontend (from RegisteredObservabilityProducerV1::producer()) still held the shared producer core open, so the next mount could start a second producer for the same store while the first still accepted and wrote events — exactly the duplicate-producer bug the store-authority registry exists to prevent.

Fix (src/daemon/service/project_runtime/observability.rs)

  • New StoreObservabilityDrainV1 { InFlight, Deferred } on the Stopping state records whether the core drain is actually running.
  • Drop for RegisteredObservabilityProducerV1 without a runtime now begins retirement as Deferred and never settles it: the entry stays Stopping, refusing all mounts, and only warns.
  • acquire_or_start hitting a Deferred retirement on a live runtime flips it to InFlight (under the registry lock, exactly once) and spawns the real core.shutdown() drain — the mount itself is still refused with Retiring.
  • Only the confirmed drain outcome settles the entry via finish_retirement: success vacates it, failure is remembered as Failed. Explicit shutdown() and runtime-present drops are unchanged (InFlight, shared spawn_retirement_drain).

Regression test (RED on pre-fix code)

runtimeless_last_alias_drop_keeps_the_store_retiring_until_the_drain_confirms: mounts a producer, retains the frontend from producer(), drops the last alias handle on a std::thread (no tokio runtime), proves the retained frontend still enqueues, then attempts an overlap mount whose start closure panics if reached.

On the pre-fix code (baa5cb8d9) the test fails with:

panicked at src/daemon/service/project_runtime/observability_tests.rs:1182:9:
a runtimeless drop must not vacate the store entry into a duplicate producer

i.e. the vacated entry let the registry invoke the start closure for a duplicate producer while the retained frontend was still live. Post-fix the overlap mount is refused Retiring, the deferred drain then confirms (retained frontend observes observability_producer_closed), and a replacement mounts and shuts down cleanly.

Receipts

  • cargo test --lib daemon::service::project_runtime::observability_tests — 10 passed, 0 failed (includes the new test)
  • cargo test -p tracedecay-usecases --lib observability — 70 passed, 0 failed
  • cargo clippy -p tracedecay --lib --tests -- -D warnings — no diagnostics in touched files; remaining errors are pre-existing in src/daemon/code_index_scheduler* (peer lane, unchanged here)
  • cargo fmt clean on touched files

A last-alias drop without a tokio runtime treated inability-to-await as
a successful retirement and vacated the exact-store entry while retained
producer frontends still held the shared core open, so a replacement
mount could start a second producer for the same store. The entry now
stays Stopping with a deferred drain; the next mount attempt on a live
runtime starts the real core drain, and only its confirmed outcome
releases (or fails) the entry.
@changeset-bot

changeset-bot Bot commented Aug 22, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: f61afbd

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@ScriptedAlchemy
ScriptedAlchemy merged commit 4757f6a into codex/tracedecay-total-redesign-plan Aug 22, 2026
8 of 13 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f61afbd024

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +260 to +261
*drain = StoreObservabilityDrainV1::InFlight;
self.spawn_retirement_drain(&runtime, Arc::clone(core));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep deferred drains retryable after task cancellation

When the first mount after a runtimeless drop runs on a short-lived or shutting-down Tokio runtime, this permanently changes the entry to InFlight before spawning a fire-and-forget drain. If that runtime cancels the task before finish_retirement, subsequent mounts continue returning Retiring and never restart the drain, wedging the store for the registry's lifetime. Retain a retryable state or track task cancellation through settlement.

AGENTS.md reference: AGENTS.md:L114-L116

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant