Skip to content

Spec: bridge ORB's submitter-reputation signal with AMS's calibration/track-record data #6208

Description

@JSONbored

Problem

ORB's src/review/submitter-reputation.ts derives a private, per-(project, submitter) quality signal purely from that repo's own review_targets history — it has no concept of a contributor's track record anywhere else. AMS separately maintains its own calibration/track-record data for a miner's own work: packages/loopover-engine/src/track-record-summary.ts, packages/loopover-miner/lib/prediction-ledger.js, and the calibration modules (gate-verdict-calibration.ts, finding-severity-calibration.ts, reviewer-consensus-calibration.ts). These two systems have never been connected: a contributor with a long, high-quality AMS track record on other repos is, from ORB's perspective on a new repo, indistinguishable from a first-time low-reputation submitter.

Area

src/review/submitter-reputation.ts (ORB), AMS's calibration/track-record stack (@loopover/engine, packages/loopover-miner/lib/prediction-ledger.js).

Proposal (open questions a design pass needs to answer — not yet decided)

  • Identity linkage: how does ORB know a given GitHub login corresponds to a specific AMS miner's track record? Is there already a stable identity (a gittensor hotkey, a GitHub login match, something else), or does this need new plumbing? submitter-reputation.ts is strictly GitHub-login-keyed today.
  • Data path: does ORB pull AMS's track-record data via a new authenticated API call (analogous to how resolveGitHubToken calls loopover's own backend), or does AMS push a signal into ORB's existing reputation store when a PR is opened? Consider which direction keeps both products' independence intact — ORB must still function with zero knowledge of AMS if AMS isn't present.
  • Weighting: should a strong AMS track record only ever move the signal toward "trusted" (never toward "low," to avoid a miner's AMS behavior being used punitively against their standing on an unrelated repo), matching submitter-reputation.ts's existing "default GENEROUS" philosophy?
  • Privacy: this must stay exactly as internal as submitter-reputation.ts already is today — no public label, comment, or check-run may ever reveal it. Confirm the AMS-side data (calibration/prediction-ledger) doesn't itself carry anything that would violate that if surfaced.

Deliverables

  • A decided design: identity-linkage mechanism, data path (pull vs. push), weighting rule, and privacy boundary.
  • A written spec (this issue's own body, updated with the decision, or a follow-up doc) that a future contributor-open implementation issue can be scoped from.

Resources

  • src/review/submitter-reputation.ts (the existing internal-only reputation system, its own header comment documents its "default GENEROUS" + strictly-internal contracts)
  • packages/loopover-engine/src/track-record-summary.ts, packages/loopover-miner/lib/prediction-ledger.js

Boundaries

  • maintainer-only until the design questions above are resolved — this is an identity/trust-signal decision, not mechanical engineering.
  • Must not create a way for a miner to game their reputation on a repo they've never contributed to.
  • ORB must continue to function with zero knowledge of AMS if AMS isn't present on a given repo (no hard dependency introduced).

Metadata

Metadata

Assignees

No one assigned

    Labels

    maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

    Projects

    Status
    Done
    Status
    Done

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions