Skip to content

db: decision_ledger_anchors has no index on `row_hash #9652

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

anchorBackendsMissingForRowHash (src/review/ledger-anchor-persistence.ts:182) was added by #9489 and runs
on every anchor-decision-ledger tick (src/review/ledger-anchor-scheduler.ts:101, dispatched from
src/queue/job-dispatch.ts:114). Its statement is:

SELECT DISTINCT backend FROM decision_ledger_anchors
 WHERE row_hash = ?1 AND status = 'ok' AND backend IN (…)

decision_ledger_anchors has exactly two indexes, both created in migrations/0195 and recreated verbatim
after the rebuild in migrations/0201: (created_at DESC) and (backend, created_at DESC). Neither leads
with row_hash, so this lookup is a full table scan of an append-only table on every tick, on both the D1 and
the self-host backends.

The same change (#9489) introduced a second new lookup with the same shape —
verifyDecisionLedger's NOT EXISTS anti-join on decision_ledger.record_id — and that one did get its
index, in migrations/0198_orb_outcome_rollups.sql:25, with an explicit comment: "without this index that is
a full ledger scan per candidate record."
The row_hash lookup added by the same issue was left unindexed.
This is a straight sibling asymmetry, not a judgement call: decision_ledger_anchors is append-only with no
retention rule in RETENTION_POLICY (src/db/retention.ts:16-108), so it only ever grows.

Requirements

  • A new contiguous migration migrations/0202_*.sql creates
    CREATE INDEX IF NOT EXISTS decision_ledger_anchors_row_hash_status ON decision_ledger_anchors (row_hash, status);
    — leading with row_hash (the equality predicate) and including status so the status = 'ok' filter is
    served from the index rather than a row fetch.
  • The migration's header comment must name anchorBackendsMissingForRowHash and ledger: four anchoring defects — unretried failures at a quiet tip, a tip-read race publishing a false tamper signal, 1MB file truncation, short_tail blind spots #9489, and state the sibling
    precedent (decision_ledger_record_id in migrations/0198), in the same style as the surrounding migrations.
  • The migration must contain nothing else — no schema change, no data change.
  • npm run db:migrations:check and npm run db:schema-drift:check must both pass
    (decision_ledger_anchors is already in RAW_SQL_ONLY_TABLES, so no src/db/schema.ts change is needed or
    wanted).

⚠️ Required pattern: mirror migrations/0198_orb_outcome_rollups.sql:23-25 — a single
CREATE INDEX IF NOT EXISTS with a comment naming the query it serves and the issue that introduced that
query. What does NOT satisfy this issue: rewriting anchorBackendsMissingForRowHash's SQL instead of adding
the index; adding the index by rebuilding the table (0201's rebuild-and-rename pattern is for CHECK
constraints only — an index needs no rebuild); adding it to an existing migration file (already applied
files must never be edited — see src/selfhost/migrate.ts:106-118's content-hash drift check); or choosing a
different column list than the one specified above.

Deliverables

  • migrations/0202_<snake_case_name>.sql exists, is the only new migration, and contains exactly the one
    CREATE INDEX IF NOT EXISTS decision_ledger_anchors_row_hash_status ON decision_ledger_anchors (row_hash, status);
    statement plus its header comment.
  • A test asserting the index exists after replaying migrations/** — e.g. a case in
    test/unit/ querying sqlite_master for
    type='index' AND name='decision_ledger_anchors_row_hash_status' against the replayed migrations DB, in
    the style of scripts/check-schema-drift.ts's replayMigrations/listActualTables helpers.
  • A test asserting anchorBackendsMissingForRowHash still returns the same results (all backends missing /
    some present / all present) after the migration, so the index cannot have changed behaviour.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example adding
the migration without the Deliverable-2 existence assertion, so a future table rebuild can silently drop the
index the way 0201 would have — does not resolve this issue.

Test Coverage Requirements

migrations/** is not measured by Codecov (coverage.include covers src/**,
packages/loopover-engine/src/**, packages/loopover-miner/{lib,bin}/**, packages/discovery-index/src/**,
packages/loopover-contract/src/** and packages/loopover-mcp/{lib,bin}/**), but
src/review/ledger-anchor-persistence.ts IS, and the Deliverable-3 tests exercise it — all three arms (none
anchored / partially anchored / fully anchored) must be covered so the patch gate is satisfied for any line
touched there.

Expected Outcome

The per-tick "which backends still lack a successful anchor for this tip" lookup is an index seek rather than a
full scan of a permanently-growing append-only table, matching the treatment its sibling lookup from the same
issue already received.

Links & Resources

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions