feat(orb): fleet calibration collector + analytics (retire per-instance App; feed from review_audit) - #1256
Merged
Merged
Conversation
Orb stops being a per-instance GitHub App and becomes a central, anonymized fleet-calibration collector fed by self-hosted instances. Each instance already records de-noised ground truth in review_audit (gate_decision + pr_outcome + reversal_reopened/reversal_reverted); the exporter now ships THAT up instead of reinventing a noisier raw merged/closed signal. - Retire the per-instance Orb App: delete orb-setup.ts + orb-webhook.ts, the /orb/setup* and /orb/webhook routes + dead counters, the orb_events/orb_installations tables, the second App secret. - Repoint exportOrbBatch to a portable review_audit join (window functions + CASE — runs on the SQLite OR Postgres self-host backend); carry the reversal flag, a bucketed reason category, and decision->close cycle time; resume from an orb_export_cursor watermark. - Extend the receiver: orb_signals re-keyed on (instance_id, repo_hash, pr_hash), upsert (OR REPLACE so a later reversal updates the row), new fields + cycle-time clamp (migration 0060). - Add fleet analytics: src/orb/analytics.ts (median-robust gate precision / FP / FN / reversal / cycle-time, outlier detection) behind a bearer-gated GET /v1/internal/fleet/analytics. Anonymization unchanged: repo/PR identifiers HMAC'd with each instance's own secret, so the collector holds no instance secret and can never de-anonymize. No shared key, no Orb App, no wizard — maintainers just flip ORB_ENABLED=true. Verified: 100% branch coverage on the orb modules, full test:ci green, migration 0060 applies on real Postgres.
Contributor
|
Superagent didn't find any vulnerabilities or security issues in this PR. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1256 +/- ##
=======================================
Coverage 94.97% 94.98%
=======================================
Files 177 176 -1
Lines 19827 19849 +22
Branches 7133 7143 +10
=======================================
+ Hits 18831 18853 +22
Misses 397 397
Partials 599 599
🚀 New features to boost your workflow:
|
10 tasks
JSONbored
added a commit
that referenced
this pull request
Jun 24, 2026
…1268) Expose the fleet calibration analytics (computeFleetAnalytics, shipped in #1256) through two operator-facing read surfaces: - gittensory_get_fleet_analytics MCP tool — operator-gated (sessions need operator authority; trusted token/static identities pass), measurement only. Lets the owner audit fleet drift from their agent harness. - Operator dashboard fleetMetrics — median merge/close precision, FP + reversal rates, and cycle-time percentiles, plus two summary cards (Fleet instances, Fleet merge precision) and a Fleet health panel in the operator route. Both are read-only aggregates over the anonymized orb_signals; no per-repo or sensitive terms cross the boundary. Advances #1255.
5 tasks
JSONbored
added a commit
that referenced
this pull request
Jul 12, 2026
…nel id, make github-prs $scope dynamic (#5302) Phase C of the observability audit: 4 gittensory.json panels referenced gittensory_orb_events_recorded_total/gittensory_orb_installs_total, both retired along with the per-instance Orb App in #1256 but never cleaned out of the dashboard -- they've rendered a permanent 0 ever since, and the locked-in test asserted the broken names as expected content. Removed the 2 panels with no live equivalent (there's no "recorded" step distinct from "exported" in the current fleet-collector architecture) and stripped the dangling series from the 2 mixed panels that partially still work. Also: renumbered a duplicate panel id (158 used by two different panels), fixed a stale alerts.yml comment claiming status-label/latency-histogram metrics are still "being added" (both are fully live), converted github-prs.json's $scope from a hardcoded 5-value custom list to a query-backed variable built from whatever repos are actually tracked in the local reporting DB, and documented (with an empirical Grafana-API check) why the grafana-github-datasource can't be locked read-only like the other 4 datasources without reintroducing the boot-crash risk its API-based provisioning was built to avoid.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the data pipeline + analytics for #1255 — Orb as the central, anonymized fleet-calibration collector fed by self-hosted instances.
What changed
Retired the per-instance Orb GitHub App (the "tons of custom work per instance" we wanted to avoid): deleted
orb-setup.ts+orb-webhook.ts, the/orb/setup*and/orb/webhookroutes + dead counters, and theorb_events/orb_installationstables. Each self-hosted instance already records de-noised ground truth inreview_auditvia the engine'srecordPrOutcome+recordReversalSignals— Orb ships that up, not a noisier reinvention.Repointed the exporter (
exportOrbBatch) to a portablereview_auditjoin (window functions +CASE, so it runs on the self-host SQLite or Postgres backend). Each exported event now carries the reversal flag (reopened/reverted— the high-value "the gate was wrong" signal Orb was missing), a bucketed reason category, and decision→close cycle time. Resumes from anorb_export_cursorwatermark.Extended the receiver + schema (migration
0060):orb_signalsre-keyed on(instance_id, repo_hash, pr_hash)(the old(instance_id, pr_hash)collided across repos),INSERT OR REPLACEupsert (a later reversal updates the row), new columns + a cycle-time clamp, and a(verdict, outcome, reversal)index.Added fleet analytics —
src/orb/analytics.ts: median-robust gate precision / false-positive / false-negative / reversal rate / cycle-time with per-instance outlier detection, behind a bearer-gatedGET /v1/internal/fleet/analytics(mirrors/v1/internal/ops/stats).Privacy / trust (unchanged guarantees)
repo/PR identifiers are HMAC'd with each instance's own secret, so the collector holds no instance secret and can never de-anonymize. No shared key, no Orb App, no wizard — maintainers just set
ORB_ENABLED=true. Aggregation is median/percentile, never mean, so one instance feeding garbage can't move the fleet numbers.Validation
npm run test:cifully green;npm audit0 vulnerabilities0060applies on real Postgres (theselfhost-pgintegration test)Advances #1255 — remaining follow-ups (kept open)
gittensory_get_fleet_analyticsMCP tool (read surface over the samecomputeFleetAnalytics)