diff --git a/apps/loopover-ui/content/docs/verify-this-review.mdx b/apps/loopover-ui/content/docs/verify-this-review.mdx index da39dd3777..1cace0e727 100644 --- a/apps/loopover-ui/content/docs/verify-this-review.mdx +++ b/apps/loopover-ui/content/docs/verify-this-review.mdx @@ -11,16 +11,27 @@ website only build trust if a skeptic can check them without asking anyone's per page is the end-to-end walkthrough: export the same corpus snapshot the numbers come from, verify its checksum, replay the same scorer over it, and compare what you get against what is published. -Everything below runs read-only against a database export and pure functions from -`@loopover/engine`. Nothing posts anywhere, nothing needs an API key. +Everything below runs read-only against a corpus export and pure functions from `@loopover/engine`. +Nothing posts anywhere, and nothing needs a LoopOver API key. + + + **Step 1 is not yet runnable by a stranger.** Exporting the corpus reads the deployment's own + database — with `--remote` that means the operator's Cloudflare D1, which needs *their* Cloudflare + credentials, not a LoopOver key. So today steps 1–3 are reproducible by an operator or by anyone + running a self-host deployment against their own data, and step 4 is the part a third party can + check unauthenticated. No anonymous corpus download exists yet; until one does, this page marks + each step with who can actually run it rather than implying everyone can run all of them. + For the wider contract — every claim, its artifact, and its trust assumption, including the ones you cannot check — see [what you can verify](/docs/what-you-can-verify). -## 1. Export the corpus snapshot +## 1. Export the corpus snapshot *(operator / self-host)* Every rule's fired/override history exports as a versioned, checksummed JSON snapshot -([backtest & calibration](/docs/backtest-calibration) explains how that history is recorded): +([backtest & calibration](/docs/backtest-calibration) explains how that history is recorded). +`--remote` shells out to `wrangler d1 execute … --remote`, so it reads the deployed database and +requires that deployment's own Cloudflare credentials: ```bash npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch --output corpus.json --remote @@ -34,10 +45,12 @@ npx tsx scripts/backtest-corpus-export.ts --rule-id linked_issue_scope_mismatch The snapshot's `checksum` field is a SHA-256 over the canonicalized cases (keys sorted, so property order can never change the hash). The fairness report's *reproducibility freeze point* -shows the checksum of the corpus behind the latest persisted backtest run — an export of the same -window reproduces the same checksum, byte for byte. +shows that checksum for the **most recent persisted backtest run**, whichever single rule that run +covered — re-exporting that rule over the same window reproduces the same checksum, byte for byte. +It is one run's freeze point, not a per-rule commitment for every rule on the report, so compare it +against an export of the rule the run actually covered. -## 2. Verify the checksum +## 2. Verify the checksum *(anyone, given a snapshot)* The manifest is self-verifying: recompute the hash over its own `cases` array and compare it to the recorded `checksum`. The canonicalization lives in `scripts/backtest-corpus-export-core.ts` @@ -53,7 +66,7 @@ console.log(recomputed.checksum === saved.checksum ? "checksum OK" : "CHECKSUM M ' ``` -## 3. Replay the scorer +## 3. Replay the scorer *(anyone, given a snapshot)* The published precision comes from the same pure functions any Node script can import: `scoreBacktest` replays a classifier over the labeled cases, and `compareBacktestScores` applies @@ -75,19 +88,42 @@ console.log(report); - **`null` is never `0`.** Precision and recall stay `null` below the decided-sample floor; the fairness report renders that as *insufficient data*, never as a zero. -## 4. Compare against the published numbers +## 4. Compare against the published numbers *(anyone)* + +The [fairness report](/fairness) renders each rule's decided-case count and measured precision from +the public stats endpoint's `rulePrecision` block. That endpoint is served by the API host, not by +this site, so fetch it absolutely — a bare `/v1/public/stats` resolves against `loopover.ai` and +404s: + +```bash +curl -s "https://api.loopover.ai/v1/public/stats" | jq '.rulePrecision' +``` + +The same per-rule numbers are also published as digest-committed +[EvalScoreRecords](/docs/what-you-can-verify), each independently re-derivable without trusting the +transport — recompute `recordDigest` over the record's own remaining fields and compare: + +```bash +curl -s "https://api.loopover.ai/v1/public/eval-scores" | jq '.records' +``` + +That array is empty whenever the latest backtest run's corpus is empty. A checksum over zero cases +is byte-identical for every rule and every window, so it commits to nothing a reader could re-derive +the scores from — publishing a record against it would assert a reproducibility that does not exist. +An empty `records` array therefore means *the numbers are not currently committed to a corpus*, never +that the numbers are zero. -The [fairness report](/fairness) renders each rule's decided-case count and measured precision -from the public stats endpoint (`/v1/public/stats`, the `rulePrecision` block). The aggregated -run history is also readable directly: +The aggregated run history is readable directly too, again against the deployment's own database +(operator / self-host): ```bash npx tsx scripts/backtest-track-record.ts --db loopover --remote ``` Your replayed `confirmed / decided` for a rule should match the published precision for the same -window; the freeze-point checksum ties the published numbers to the exact corpus you just -verified. +window. The freeze-point checksum ties the published numbers to the corpus you just verified only +for the rule the latest run actually backtested — for any other rule it is a timestamped pointer to +a different run, not a commitment to that rule's own cases. ## 5. Verify an attested run (when a run carries one) diff --git a/apps/loopover-ui/content/docs/what-you-can-verify.mdx b/apps/loopover-ui/content/docs/what-you-can-verify.mdx index 3df962efc3..6fa1d12b71 100644 --- a/apps/loopover-ui/content/docs/what-you-can-verify.mdx +++ b/apps/loopover-ui/content/docs/what-you-can-verify.mdx @@ -176,7 +176,10 @@ replayable history — not hand-entered numbers. This is the walkthrough in [Verify this review](/docs/verify-this-review): export the checksummed corpus, verify its checksum, re-run the same public scoring functions from `@loopover/engine`, and -compare. **Anyone** can do this for public repositories. +compare. The checksum verification and the scorer replay are pure functions **anyone** can run over +a snapshot they hold, and the published numbers themselves are fetchable unauthenticated — but the +export step that produces the snapshot reads the deployment's own database and needs that +deployment's credentials, so it is an operator / self-host step today rather than an anonymous one. **Private repositories are the exception.** A hosted tenant's review history cannot be published diff --git a/src/review/eval-score-records.ts b/src/review/eval-score-records.ts index 64e9f9305f..c1b6520764 100644 --- a/src/review/eval-score-records.ts +++ b/src/review/eval-score-records.ts @@ -4,7 +4,7 @@ // It adds no new scoring and no new trust -- the numbers are the same ones `/v1/public/stats` already // publishes, just committed to a corpus checksum and made independently re-derivable per-record. import { canonicalJson, contentDigest } from "./decision-record"; -import type { PublicRulePrecision } from "./public-rule-precision"; +import { EMPTY_CORPUS_CHECKSUM, type PublicRulePrecision } from "./public-rule-precision"; export const EVAL_SCORE_RECORD_SCHEMA_VERSION = 1 as const; @@ -76,6 +76,14 @@ async function finalizeRecord(input: EvalScoreRecordDigestInput): Promise { if (!precision.latestBacktestRun) return []; const { corpusChecksum } = precision.latestBacktestRun; + if (corpusChecksum === EMPTY_CORPUS_CHECKSUM) return []; const windowStart = new Date(Date.parse(issuedAt) - precision.windowDays * 24 * 60 * 60 * 1000).toISOString(); const records = await Promise.all( @@ -145,3 +154,4 @@ export async function verifyEvalScoreRecordDigest(record: EvalScoreRecord): Prom // Re-exported so callers that only import this module never need a second import from decision-record.ts // just to canonicalize something alongside a record (e.g. logging, or a future signed-bundle wrapper). export { canonicalJson, contentDigest }; +export { EMPTY_CORPUS_CHECKSUM }; diff --git a/src/review/public-rule-precision.ts b/src/review/public-rule-precision.ts index 18b7d9cc7a..c5136cf3f1 100644 --- a/src/review/public-rule-precision.ts +++ b/src/review/public-rule-precision.ts @@ -23,6 +23,15 @@ export const PUBLIC_PRECISION_MIN_DECIDED = 10; // duplication rule-calibration-trend.ts documents for its identical queries. const HUMAN_OVERRIDE_EVENT_TYPE_PREFIX = "signal.human_override:"; +/** `checksumCases([])` — SHA-256 over the canonicalized empty case list (the two-byte string `"[]"`), i.e. what + * a corpus export produces for a rule with no labeled cases at all. A hash over zero cases is the same 32 + * bytes for every rule, every window and every deployment, so it is not the "independently-verifiable freeze + * point" {@link PublicRulePrecision.latestBacktestRun} claims to be — it points at nothing a skeptic could + * re-derive anything from. Hard-coded because the canonicalization that produces it + * (`scripts/backtest-corpus-export-core.ts`) runs on `node:crypto` and is unimportable from the Workers + * runtime; this module's own test re-derives it from `sha256Hex("[]")` so it can never drift. */ +export const EMPTY_CORPUS_CHECKSUM = "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945"; + export type PublicRulePrecisionRow = { ruleId: string; decided: number; @@ -40,7 +49,9 @@ export type PublicRulePrecision = { /** All three reversal shapes counted over the window — the "counted against ourselves" number. */ reversals: { reopened: number; reverted: number; superseded: number }; /** The latest persisted backtest run carrying a corpus checksum — the independently-verifiable freeze - * point — or null when no run has been recorded yet. */ + * point — or null when no run has been recorded yet, or when the latest run's corpus was empty (see + * {@link EMPTY_CORPUS_CHECKSUM}: a commitment to nothing is not a freeze point, so it is reported as + * absent rather than published as though it were verifiable). */ latestBacktestRun: { corpusChecksum: string; at: string } | null; }; @@ -102,6 +113,9 @@ export async function loadPublicRulePrecision(env: Env, nowMs: number = Date.now reverted: reversalCount("reversal_reverted"), superseded: reversalCount("reversal_superseded"), }, - latestBacktestRun: latest && typeof latest.checksum === "string" && latest.checksum !== "" ? { corpusChecksum: latest.checksum, at: latest.created_at } : null, + latestBacktestRun: + latest && typeof latest.checksum === "string" && latest.checksum !== "" && latest.checksum !== EMPTY_CORPUS_CHECKSUM + ? { corpusChecksum: latest.checksum, at: latest.created_at } + : null, }; } diff --git a/test/unit/eval-score-records.test.ts b/test/unit/eval-score-records.test.ts index 1b8479ddf1..910e58ca3f 100644 --- a/test/unit/eval-score-records.test.ts +++ b/test/unit/eval-score-records.test.ts @@ -6,9 +6,10 @@ import { ORB_GATE_SUBJECT_ID, OUTCOME_CONFIRMED_PRECISION_SCORING_RULE_VERSION, verifyEvalScoreRecordDigest, + EMPTY_CORPUS_CHECKSUM, type EvalScoreRecord, } from "../../src/review/eval-score-records"; -import { contentDigest } from "../../src/review/decision-record"; +import { contentDigest, sha256Hex } from "../../src/review/decision-record"; import type { PublicRulePrecision } from "../../src/review/public-rule-precision"; const ISSUED_AT = "2026-07-27T12:00:00.000Z"; @@ -29,6 +30,25 @@ describe("buildEvalScoreRecordsFromRulePrecision (#9266)", () => { expect(records).toEqual([]); }); + it("refuses to publish records whose freeze point commits to an empty corpus", async () => { + // Regression: production published decided=460/confirmed=287 alongside sha256("[]") -- a hash that is + // byte-identical for every rule and every window, so it committed to nothing a consumer could re-derive. + const records = await buildEvalScoreRecordsFromRulePrecision( + { ...PRECISION_WITH_FREEZE_POINT, latestBacktestRun: { corpusChecksum: EMPTY_CORPUS_CHECKSUM, at: "2026-07-27T10:00:00.000Z" } }, + ISSUED_AT, + ); + expect(records).toEqual([]); + }); + + it("EMPTY_CORPUS_CHECKSUM is the exporter's own checksum over zero cases", async () => { + // scripts/backtest-corpus-export-core.ts hashes `JSON.stringify(cases.map(canonicalizeCase))`, which for + // an empty list is the two-byte string "[]" -- re-derived here so the hard-coded constant cannot drift + // from the exporter that produces the value it guards against. (That exporter's canonicalization and + // canonicalJson coincide ONLY on the empty case, so this hashes the literal preimage rather than + // round-tripping [] through either one.) + expect(EMPTY_CORPUS_CHECKSUM).toBe(await sha256Hex("[]")); + }); + it("builds one record per rule, committed to the freeze point's corpus checksum", async () => { const records = await buildEvalScoreRecordsFromRulePrecision(PRECISION_WITH_FREEZE_POINT, ISSUED_AT); expect(records).toHaveLength(2); diff --git a/test/unit/public-rule-precision.test.ts b/test/unit/public-rule-precision.test.ts index fdf4e5d85d..085c028dba 100644 --- a/test/unit/public-rule-precision.test.ts +++ b/test/unit/public-rule-precision.test.ts @@ -3,7 +3,9 @@ import { loadPublicRulePrecision, PUBLIC_PRECISION_MIN_DECIDED, PUBLIC_PRECISION_WINDOW_DAYS, + EMPTY_CORPUS_CHECKSUM, } from "../../src/review/public-rule-precision"; +import { sha256Hex } from "../../src/review/decision-record"; import { recordAuditEvent } from "../../src/db/repositories"; import { createSignalStore } from "../../src/review/signal-tracking-wire"; import { createTestEnv } from "../helpers/d1"; @@ -95,6 +97,33 @@ describe("loadPublicRulePrecision (#8230)", () => { expect(block.latestBacktestRun).toEqual({ corpusChecksum: "newest111", at: new Date(NOW - 30_000).toISOString() }); }); + it("reports no freeze point when the latest run's corpus was empty", async () => { + // Regression: production's latest run carried sha256("[]") -- the checksum of ZERO cases -- and the + // fairness page rendered it as a "reproducibility freeze point" while /v1/public/eval-scores published + // records committed to it. A hash over no cases is identical everywhere, so it verifies nothing. + const env = createTestEnv(); + await seedVerdicts(env, "ai_consensus_defect", 15, 5); + await recordAuditEvent(env, { + eventType: "calibration.logic_backtest_run", + targetKey: "rule", + outcome: "completed", + metadata: { corpusChecksum: EMPTY_CORPUS_CHECKSUM, comparison: {} }, + createdAt: new Date(NOW - 1000).toISOString(), + }); + + const block = await loadPublicRulePrecision(env, NOW); + expect(block.latestBacktestRun).toBeNull(); + // The scores come from a different dataset (human-override events) and are unaffected -- an empty corpus + // means the numbers are uncommitted, never that they are zero. + expect(block.rules).toEqual([{ ruleId: "ai_consensus_defect", decided: 20, confirmed: 15, precision: 0.75 }]); + }); + + it("EMPTY_CORPUS_CHECKSUM is the exporter's own checksum over zero cases", async () => { + // scripts/backtest-corpus-export-core.ts hashes `JSON.stringify(cases.map(canonicalizeCase))`, which for + // an empty list is the two-byte string "[]" -- re-derived here so the constant cannot drift from it. + expect(EMPTY_CORPUS_CHECKSUM).toBe(await sha256Hex("[]")); + }); + it("degrades fail-safe on a broken store and reports null freeze point on a fresh ledger", async () => { const empty = await loadPublicRulePrecision(createTestEnv(), NOW); expect(empty).toEqual({