Skip to content

Add deterministic agent context compiler - #6503

Closed
salman1993 wants to merge 1 commit into
mainfrom
codex/context-compiler-manifest
Closed

Add deterministic agent context compiler#6503
salman1993 wants to merge 1 commit into
mainfrom
codex/context-compiler-manifest

Conversation

@salman1993

@salman1993 salman1993 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Why

Agent adapters need a deterministic, auditable boundary between durable standing context, prior conversation, and the event authorizing the current turn.

What

  • Compile dynamic turns into a Buzz-owned semantic envelope with <ambient-context>, a current <event author="…">, and <delivery>, while standing sections use paired domain tags such as <workspace>, <base>, and <system>
  • Keep model-visible framing minimal: section position conveys instruction authority, nonessential source metadata remains manifest-only, and raw events stay available through buzz messages raw
  • Publish canonical hashed context-layer manifests, report channel metadata and session-start canvas freshness separately, and incorporate accepted native steers into the completed turn record
  • Escape arbitrary standing-context bodies so embedded closing tags cannot spoof boundaries; Desktop diagnostics decode them after parsing and retain legacy prompt-history compatibility

Risk Assessment

Medium — this changes ACP prompt serialization and turn telemetry, but preserves context ordering, keeps legacy observer parsing, and requires no relay storage migration.

Visualizing the change

Screenshot 2026-08-24 at 12 38 18 PM

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
@salman1993

Copy link
Copy Markdown
Contributor Author

Benchmark complete. Recommendation: do not adopt PR #6503 as-is. Its efficiency gains do not preserve important Buzz-native behavior.

Suite Baseline Candidate Outcome
Terminal-Bench 22/22 20/22 −2 passes
Buzz-native 12/18 8/18 −4 passes

Terminal candidate improvements were substantial: active time −32%, wall time −26%, input/output tokens −30%, and estimated cost −34%. Buzz-native efficiency changes were small (roughly 2–7%) and likely noise.

The native regression was concentrated in:

  • cross-thread-requests: 2/2 → 0/2; correct answers and thread anchors, but missing human mention tags.
  • multiline-message: 1/2 → 0/2; mentions omitted, with one reply posted top-level.
  • narrative-agent-names: 1/2 → 0/2; callback mention/thread destination split incorrectly.
  • Channel TTL, named-path reads, interleaved reports, and membership behavior were preserved.

@salman1993 salman1993 closed this Aug 24, 2026
@salman1993

salman1993 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

Update: lower active-h is good, but the original Terminal-Bench improvement was partially inflated by two short failed candidate runs.

Looking only at matched attempts where both conditions passed:

Suite Baseline active Candidate active Change
Terminal-Bench, 20 matched passes 1,128.9 s 879.5 s −22.1%
Buzz-native, 8 matched passes 126.6 s 126.1 s −0.4%

For Terminal-Bench, the candidate was faster in 17/20 matched pairs. Model time fell 24.7%, output tokens fell 25.2%, and every matched run had a completion message, no exception, reward 1.0, and no failed verifier subchecks. That supports a genuine shorter successful trajectory—not an early runner exit.

The all-trial Terminal figure was −32.4%, rather than −22.1%, because:

  • Failed candidate pytorch-model-recovery: 64.6 s versus passing baseline 245.1 s.
  • Failed candidate regex-log: 90.1 s versus passing baseline 154.9 s.

So the accurate conclusion is:

  • Terminal-Bench: candidate is genuinely faster on common successes, around 22% by summed active time.
  • Buzz-native: no credible speed improvement after matching passes.
  • Correctness regressions still prevent recommending the candidate as-is.

salman1993 added a commit that referenced this pull request Aug 26, 2026
## Why
PR #6503 bundled a broader context compiler with several prompt-shape
and telemetry changes, making it difficult to isolate whether paired
XML-like top-level boundaries improve agent behavior. This keeps the
follow-up experiment focused on framing only.

## What
- Replace top-level standing and per-turn bracket headers with paired
tags while preserving inner section content and event/message framing
- Carry only metadata already present in dynamic headers as attributes,
such as thread counts, truncation, event type, and batch count
- Preserve every section body verbatim so the model executes the exact
bytes reviewed in Desktop
- Keep routing metadata, event content, batching, and delivery behavior
unchanged
- Teach Desktop diagnostics to parse the new framing while retaining
compatibility with archived bracket-framed sessions; ambiguous paired
boundaries render as the complete captured prompt

## Risk Assessment
Medium — the change is limited to agent prompt framing and diagnostic
parsing, but it affects both standing context and every dynamic agent
turn. Bodies and delivery behavior remain unchanged, and compatibility
parsing remains for prior sessions.

## References
- Follow-up to #6503, scoped only to the paired-tag framing hypothesis

## Update — 2026-08-24
- Addressed review feedback by removing entity encoding/decoding from
the execution and diagnostic paths.
- Added regressions for literal closing tags, placeholders,
entity-looking text, ampersands, and authored boundary whitespace.
- Verified `cargo test -p buzz-acp` (805 unit + 9 integration), Desktop
tests (5,412 before the main sync; 5,439 in the pre-push gate), Desktop
typechecking, and Tauri clippy/tests.
- Live local relay startup and message ingestion passed. The live ACP
reply step is blocked locally because installed Goose 1.0.23 no longer
provides the documented `goose acp` command and no alternate ACP adapter
is installed.

---

**Update Aug 25, 12:13 EDT:** Extended paired tags to per-turn top-level
sections.

- Converted context, conversation/thread context, event batches, and
steer/interrupt boundaries without changing their inner bodies.
- Preserved existing header metadata as attributes only; added no new
prompt metadata.
- Verified 806 Buzz ACP unit tests plus 9 integration tests, 5,441
Desktop tests, Desktop typechecking and formatting, and all differential
pre-push lanes.

---

**Update Aug 25, 13:58 EDT:** Addressed framing review feedback.

- Updated operational base instructions to reference the emitted
`<context>` section and added a regression guard against stale
`[Context]` instructions.
- Restored batched steer/interrupt cardinality in Desktop diagnostic
titles with focused coverage for both paths.
- Verified 807 Buzz ACP unit tests plus 9 integration tests, 5,442
Desktop tests, Desktop typechecking and formatting, and every
differential pre-push lane including Tauri clippy/tests.

---

**Update Aug 25, 15:37 EDT:** Hardened legacy transcript compatibility.

- Semantic standing-context parsing now requires paired framing at the
start and complete input consumption.
- Historical bracket-framed personas keep literal balanced tag examples
in their original section; malformed or mixed current framing displays
as the complete captured prompt.
- Added regressions for the reported legacy persona case and trailing
unframed text; verified 60 focused parser tests, 5,444 full Desktop
tests, formatting, and typechecking.

---

**Update Aug 25, 16:31 EDT:** Preserved slash-command prompt block
boundaries.

- Desktop now sets aside a valid connector-facing slash-command block
only when the following block begins recognized paired framing, parses
the semantic blocks normally, and retains the command as visible Prompt
metadata.
- Added an end-to-end transcript regression using the production
three-block ACP shape, including the user bubble, author pubkey, event
ID, and grouped prompt context.
- Verified 113 combined transcript tests, 5,445 full Desktop tests,
formatting, typechecking, and all applicable pre-push lanes.

Generated with Codex

---------

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
salman1993 added a commit that referenced this pull request Aug 26, 2026
## Why
This is a small isolated follow-up to
[#6503](#6503): remove two proven
duplicate prompt fragments without adopting its broader framing
experiment or changing Buzz-native routing and identity behavior.

## What
- Remove parsed `parent=<id>` only when it exactly duplicates parsed
`root=<id>`; retain `root=<id>`, nested parents where parent differs
from root, raw event tags, event IDs, author identity, mention fields
and rules, and the explicit `--reply-to` instruction.
- Remove the full-history retrieval hint only when complete, untruncated
conversation or thread context is included. Thread context must include
its root and cover every event rendered in the batch; mixed-thread
batches and root-missing windows retain retrieval guidance. Context that
is absent, previously delivered, or truncated also retains retrieval
guidance, and the context header, count, and messages remain unchanged.

## Risk Assessment
Low — the change is confined to Buzz ACP prompt rendering and preserves
all routing, identity, event, and retrieval information outside the two
exact duplicate cases.

## Testing
- `cargo test -p buzz-acp` — 804 unit tests and 9 lifecycle integration
tests passed on the clean rerun
- Focused mixed-thread, same-thread, and missing-root prompt regressions
passed
- The timing-sensitive `keepalive_resets_idle_past_deadline` test passed
in isolation after one unrelated full-suite flake, then passed in the
clean full rerun
- `cargo clippy -p buzz-acp --all-targets -- -D warnings`
- `cargo fmt --all -- --check`
- `just file-size-check`
- Repository pre-push gate at `1b88793574e74e22f00a1fd19bfe9aa8e86150fa`
— file-size, branch, Rust unit-test, and Desktop Tauri lanes passed
- `git diff --check`

Generated with Codex

Signed-off-by: Salman Mohammed <smohammed@squareup.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant