Skip to content

docs(miner-selfimprove): self-improve phase design note — calibration, value-weighting, anti-farming boundaries #4265

Description

@JSONbored

Phase 7's contributor-safe half (this batch's other items — the drift detector, calibration dashboard, prediction ledger, metrics exporter, extension panel, and the confusion-matrix regression tests) is deliberately scoped to read-only measurement and detection. The maintainer-only milestone "Miner — Governor, Loop & Calibration (maintainer)" holds the sibling items that actually act on calibration data, and a contributor building against this batch needs to understand where that line is and why, or they'll naturally reach for "just wire the dashboard's accuracy number into an autonomy-level bump" — which is explicitly the maintainer-only side of this phase.

This design note should document, for a reader who only has the contributor-safe tools available:

  • What calibration measures. The Phase7CalibrationLoopResult contract (packages/gittensory-engine/src/phase7-calibration-loop.ts) combining historical_replay and pr_outcome signals against the documented 0.62 baseline (phase7-calibration-loop.ts:11), and how computeGateEval's confusion matrix (src/review/parity.ts:33-58) is the ground-truth source the live side of that measurement reads.
  • Value-weighting (maintainer: value-weighted gate-prediction calibration — optimize for merged-net-positive, not raw volume #2348, maintainer-only, already-merged prerequisite this doc should explain rather than reopen): why raw merge/close precision isn't the real objective — computeGateEval already discounts a prediction's credit when the outcome was later reversed (REVERSAL_DISCOUNT_WEIGHT, src/review/parity.ts:78, weightedMergeConfirmed/weightedCloseConfirmed), so a contributor reading a dashboard number should understand it's measuring durable correctness, not volume.
  • Personalized tuning (maintainer: personalized gate-prediction tuning per contributor/miner history #2349, maintainer-only, open): explain that per-actor confidence adjustment based on calibration history exists as a maintainer-only concept, and why a contributor-safe tool must never read/write it directly — a contributor-facing surface that could see or influence its own personalized-trust score would defeat the point of it.
  • Fleet anti-farming (maintainer: fleet-wide anti-farming detector — flag a miner/fleet gaming the calibration score #2350, maintainer-only, open): explain the gaming vectors this whole phase has to be designed around from day one — inflating merge-precision with trivially-safe PRs, or farming duplicate-issue-claim elections — so that every contributor-safe tool in this batch (the dashboard, the exporter, the ledger) is documented as read-only/local-only because a write-capable version of any of them would be a farming vector, not just an arbitrary restriction.
  • Calibration-gated circuit-breaking (maintainer: wire calibration accuracy into the live auto-tune circuit-breaker for miner-originated PRs #2352, maintainer-only, open): explain that src/review/auto-tune.ts's existing self-tightening safety breaker (holds-only, never loosens) is the eventual consumer of miner-sourced calibration data via computeGateEval's source scoping — again, enforcement is maintainer-only; the contributor-safe tools in this batch only ever produce read-only signal for a human/maintainer-owned system to eventually act on.

Deliverables

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:featureGittensor-scored feature linked to a feature issue — scores a 0.25x multiplier.gittensor:priorityMaintainer-selected Gittensor priority — scores a 1.5x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions