Skip to content

counterfactual: advisory check when reviewer prompt versions change #8222

Description

@JSONbored

Problem

Counterfactual sub-epic phase 4 (capstone). With the harness proven, close the loop the same way #8138 did for thresholds: a PR that bumps a reviewer prompt version gets an advisory comparison — old prompt vs new — replayed against the fixture set, surfaced on the PR, never blocking.

Requirements

⚠️ Required pattern. Mirror the #8138/#8139 advisory posture end-to-end: path-filtered trigger on the prompt-version constants, update-in-place PR comment via the shared renderer, persisted calibration.counterfactual_backtest_run events feeding the SAME track-record aggregation, fail-open on every operational error (a broken replay must never red a PR — the #8140-era advisory-CI lesson).

  • Spend guard: the check runs only when the deployment has an explicit counterfactual budget flag set; otherwise it posts a 'skipped: no replay budget configured' notice.
  • Sampling per the design contract keeps per-PR cost bounded and deterministic per head SHA.

Deliverables

  • Trigger + comment + persistence + track-record inclusion, 100% branch on touched src
  • Docs: the prompt-change workflow on the backtest-calibration page

Links & Resources

The sub-epic (parent), the harness issue (blocked-by), #8138/#8139 (the posture), scripts/backtest-logic-check-core.ts

Boundaries

Advisory forever until a separate, explicit authority decision (the #8105 pattern). Budget-gated by default-off flag.

maintainer-only — CI + spend wiring.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions