Orb: fleet calibration collector + analytics (retire per-instance App; feed from review_audit)

[JSONbored]

Summary / decision

Orb stops being a per-instance GitHub App and becomes a central, anonymized fleet-calibration collector + analytics layer. Gittensory cloud stays the owner's review app for our own repos; Orb aggregates de-noised calibration telemetry up from maintainers' self-hosted instances.

The key correction: every self-hosted instance already records rich, de-noised ground truth in review_audit (migration 0049) via processGitHubWebhook → recordPrOutcome + recordReversalSignals:

• gate_decision — the prediction (merge/close/hold)
• pr_outcome — ground truth (merged/closed), author-self-close-filtered
• reversal_reopened / reversal_reverted — the human-override signals

Orb's current orb_events model is a noisier reinvention of this (raw merged/closed, no reversals, broken review_targets verdict lookup, wrong head_sha dedup key) and the /orb/setup per-instance App wizard is exactly the per-instance custom work we want to avoid. So we retire that, repoint the exporter at review_audit, extend the collector schema + analytics, and keep the anonymized fleet-export (one central store, instances contribute, no shared secret — the das-github-mirror principle).

PR #1254 (merged) already shipped the live collector URL (https://gittensory-api.aethereal.dev/v1/orb/ingest), the strict per-IP rate-limit on /v1/orb/ingest, and the no-shared-secret .env.example framing. This work builds on it.

1. What Orb collects (one anonymized row per resolved PR)

Sourced from review_audit (+ review_targets for open-time), not orb_events:

Field	Source	Anonymization
instance_id	SHA256(ORB_APP_ID).slice(0,16)	one-way hash, no PII
repo_hash	HMAC(repo_full_name, ORB_WEBHOOK_SECRET).slice(0,24)	keyed by the instance's own secret → collector cannot de-anonymize
pr_hash	HMAC("${repo}#${pr}", secret).slice(0,24)	same
gate_verdict	gate_decision.decision (merge/close/hold)	none (non-PII)
outcome	pr_outcome.decision (merged/closed)	none
reversal_flag	reversal_reopened/reversal_reverted → reopened/reverted/none	none
gate_reasoncode_bucket	bucketed from gate_decision.summary reasonCode	bucketed at source; raw never leaves
decision_timestamp, outcome_timestamp	gate_decision.created_at, pr_outcome.created_at	none
time_to_close_ms	derived (see below)	none

Never exported: head_sha, raw repo/PR identifiers, actor logins, titles, code, raw reasonCode.

reasonCode buckets: issue_policy, duplicate_risk, ai_quality, slop_advisory, author_policy, ci_readiness, none, other.

Cycle-time gap (must solve, not hand-wave): time_to_close_ms currently comes only from the retiring orb-webhook.ts. review_audit has the close timestamp but not PR-open time. Resolution: join pr_outcome → review_targets.created_at (migration 0050, indexed) on target_id → time_to_close_ms = outcome_at − review_targets.created_at; fall back to outcome_at − decision_at (tagged) when no review_targets row; receiver tolerates NULL.

Dedup key: (instance_id, repo_hash, pr_hash) UPSERT (latest wins). The current 0058 UNIQUE(instance_id, pr_hash) is wrong — two instances reviewing the same owner/repo#123 collide.

Exporter join (over review_audit, mirrors computeGateEval): latest gate_decision per target_id via MAX(created_at) LEFT JOIN latest pr_outcome LEFT JOIN reversal marker LEFT JOIN review_targets. Filter source='gittensory-native'; join key is target_id (project#pr), NOT head_sha.

2. How gittensory uses it (fleet analytics)

Metric	Formula	Feeds
merge precision	merge∧merged / merge	auto-tune
close precision	close∧closed / close	auto-tune
false-positive rate	merge∧closed / merge	auto-tune + alerts
false-negative rate	close∧merged / close	observability
reversal rate	reversal≠none / decided, split reverted vs reopened	observability + trust
cycle-time p50/p95	percentiles of time_to_close_ms per instance/fleet/bucket	observability
verdict skew	per-bucket outcome distribution per verdict	alerts
instance outliers	per-instance metric vs fleet median	health dashboard

Aggregation is median/percentile, never mean (untrusted contributed data). Auto-tuning only consumes the robust, high-volume metrics (precision/FP); reversal/FN/cycle-time/skew/outliers stay advisory until per-instance reputation weighting (v2).

Surface (3 reads):

• GET /v1/internal/fleet/analytics — bearer-gated with INTERNAL_JOB_TOKEN (mirror /v1/internal/ops/stats). The ingest endpoint stays unauthenticated + rate-limited; analytics is owner-only.
• gittensory_get_fleet_analytics MCP tool (alongside gittensory_get_outcome_calibration).
• A fleetMetrics section on the operator dashboard → "Fleet Health" panel in gittensory-ui.

3. Build plan (ordered, file-level)

Migration 0060_orb_fleet_collector.sql (next contiguous after 0059):

• DROP TABLE orb_events; DROP TABLE orb_installations;
• Recreate orb_signals with the final schema (drop+recreate is clean — young regenerable telemetry table; SQLite can't ALTER away the old table-level UNIQUE): columns incl. reversal_flag (CHECK in none/reopened/reverted), decision_timestamp, outcome_timestamp, gate_reasoncode_bucket; UNIQUE(instance_id, repo_hash, pr_hash); indexes (instance_id, gate_verdict, outcome, reversal_flag) and (instance_id, received_at DESC).
• orb_export_cursor(instance_hash PK, last_exported_at, updated_at) — replaces orb_events.exported_at (review_audit is append-only).

Retire (delete): src/selfhost/orb-setup.ts, src/selfhost/orb-webhook.ts, their tests, the /orb/setup, /orb/setup/callback, /orb/webhook routes + orb_setup_state cookie in src/server.ts (≈285-333), and the dead gittensory_orb_webhook_total/gittensory_orb_installs_total counters. Keep the orbEnabled/exportOrbBatch import + hourly export cron.

Repoint (src/selfhost/orb-collector.ts): replace SELECT * FROM orb_events with the review_audit+review_targets join; drop recordOrbEvent/OrbRow/OrbEvent (local recorder gone — pr_outcome already recorded by outcomes-wire.ts); add bucketReasonCode; extend payload with new fields; replace exported_at marking with orb_export_cursor. Keep HMAC anonymization + instance_id as-is.

Extend receiver (src/orb/ingest.ts): new fields + validation (whitelist reversal_flag); INSERT OR REPLACE over the new dedup key; per-instance hourly cap + time_to_close_ms floor(1s)/ceiling(1y) + sent_at skew rejection.

Analytics: new src/orb/analytics.ts (computeFleetAnalytics, median-robust); GET /v1/internal/fleet/analytics; gittensory_get_fleet_analytics MCP tool; fleetMetrics on operator dashboard.

Config (.env.example): remove ORB_APP_ID, ORB_APP_SLUG, ORB_PRIVATE_KEY, ORB_SETUP_OUTPUT_PATH; keep/clarify ORB_ENABLED, ORB_COLLECTOR_URL, ORB_ANONYMIZE, ORB_AIR_GAP, ORB_WEBHOOK_SECRET (now the instance's own stable HMAC secret, no second App).

Tests + artifacts: repoint/extend test/unit/selfhost-orb-collector*, test/integration/orb-ingest.test.ts (new fields, invalid reversal_flag, OR REPLACE upsert, caps, cycle-time fallback both branches); both arms of every ??/ternary for the 97% branch floor; npm run ui:openapi; npm run test:coverage unsharded; npm run cf-typegen if bindings change.

4. Privacy & trust

• repo_hash/pr_hash are HMACs keyed by the instance's own secret; the collector holds no instance secret → cannot reverse. instance_id is a one-way hash. reasonCode bucketed at source. head_sha/raw names/logins/titles/code never exported.
• Image bakes zero secrets (fix(orb): point collector at the live receiver + rate-limit ingest + document the multi-tenant security model #1254). Each operator's HMAC key is unique + local; export carries no shared key. A single leaked container can't de-anonymize others, compromise the collector, or touch the main App.
• Untrusted-data robustness: median/percentile aggregation; per-instance dedup; per-IP strict rate limit; input caps. v2: per-instance reputation weighting.

What the owner does

1. Land this → migration 0060 runs on deploy (deploy:api).
2. INTERNAL_JOB_TOKEN already set (ops/stats) — no new secret.
3. New maintainers skip /orb/setup entirely — set ORB_ENABLED=true (+ a stable ORB_WEBHOOK_SECRET). No Orb GitHub App, no wizard.

Acceptance criteria

• orb_events/orb_installations + /orb/setup* + /orb/webhook removed; per-instance Orb App no longer required.
• exportOrbBatch reads review_audit join; payload carries reversal_flag + reason bucket + cycle-time (with review_targets fallback).
• orb_signals re-keyed + extended (migration 0060); receiver validates new fields + caps.
• GET /v1/internal/fleet/analytics + gittensory_get_fleet_analytics MCP tool + operator-dashboard Fleet Health section.
• Median-robust aggregation; precision/FP feed auto-tune, rest advisory.
• Full local gate green (npm run test:ci, unsharded coverage ≥97% patch, npm audit), ui:openapi regenerated.