feat(gate): #554 false-positive telemetry for hard-blocked PRs (measurement-only)

[Crystora]

Summary

Adds gate false-positive telemetry (#554). When the gittensory gate hard-blocks a PR, maintainers have no measurement of how often those blocks were wrong — i.e. the PR was later merged anyway. This is the evidence maintainers need before promoting any gate from advisory to hard-block.

This PR records every hard block and correlates it with the PR's eventual outcome, then exposes a per-gate-type false-positive rate. It is measurement-only: it never changes gate behavior, never blocks, and all writes are best-effort so a telemetry failure can never disrupt gate or webhook processing.

• gate_outcomes table (migrations/0037_gate_outcomes.sql, schema in src/db/schema.ts): one row per (repo_full_name, pr_number) recording the blocker codes and block time, plus a nullable resolution (merged / overridden) and resolved time.
• Recording (src/services/gate-telemetry.ts, src/db/repositories.ts): a hard block (conclusion: "failure") upserts an outcome; a re-block clears any prior resolution. A blocked PR that is later merged resolves the outcome to merged — a confirmed false positive. A plain close is a true positive (the block held) and is left unresolved.
• Processor hooks (src/queue/processors.ts): two straight-line, best-effort calls — one on PR close/merge, one after the gate evaluation finalizes. All branching lives in the unit-tested helpers, not the deep webhook processor.
• Read API (src/api/routes.ts): GET /v1/internal/repos/:owner/:repo/gate-telemetry returns the aggregated report — overall blocked / false-positive counts and rate, plus a per-blocker-code breakdown. Internal-auth only.

Closes #554.

Scope

• The PR title follows type(scope): short summary Conventional Commit format — feat(gate): #554 false-positive telemetry for hard-blocked PRs (measurement-only).
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes. Backend-only.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue (feat(github-app): gate false-positive telemetry #554).

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; global coverage stays ≥ 97% for lines, statements, functions, and branches (Statements 98.64%, Branches 97.04%, Functions 97.03%, Lines 99.26%)
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate (0 vulnerabilities)
• New behavior has unit + integration tests covering new branches, the best-effort write-failure fallback, and the endpoint's unauthorized path

All required checks pass via npm run test:ci (EXIT 0).

Safety

• No secrets, wallet details, hotkeys, coldkeys, PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed. The report contains only aggregate counts and blocker codes, served behind internal auth.
• Public GitHub text stays sanitized and low-noise; no compensation guarantees or optimization tactics implied. No public-facing text added.
• Auth change: the new endpoint requires internal auth and has a negative-path test asserting 401 for unauthorized requests.
• API/OpenAPI behavior is updated and tested where needed (new internal endpoint, integration-tested).
• UI changes use live API data — N/A, no UI changes.
• Visible UI changes include UI Evidence — N/A, backend-only.
• Public docs/changelogs unchanged (not a release-prep PR).

UI Evidence

N/A — backend-only change (a gate_outcomes table, recording helpers, and an internal read endpoint). No UI, frontend, docs, or extension changes.

Notes

• Measurement-only by design. No gate decision, threshold, or blocking path is altered; this PR only observes and aggregates outcomes.
• Best-effort writes. Both recording helpers wrap their DB writes in try/catch so a telemetry failure returns silently and never disrupts gate or webhook processing; a unit test asserts this via a throwing DB stub.
• False-positive definition. A block is counted as a false positive only when the PR is subsequently merged. A plain close leaves the outcome unresolved (true positive); a re-block clears any prior resolution so a reopened-then-reblocked PR is re-evaluated from scratch.
• Migration number. Uses 0037, the next sequential number after main's 0036.

[ghost]

Note

Gittensory Gate skipped

PR closed before full evaluation. No late first comment was created.

Signal	Result	Evidence	Action
Gate result	⚠️ Skipped	#755 is no longer open.	No action.

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[ghost]

gittensory · advisory review

Reviewed 9 changed file(s) — two independent AI reviewers.

Changed files (9)

• migrations/0041_gate_outcomes.sql
• src/api/routes.ts
• src/db/repositories.ts
• src/db/schema.ts
• src/queue/processors.ts
• src/services/gate-telemetry.ts
• src/types.ts
• test/integration/api.test.ts
• test/unit/gate-telemetry.test.ts

Suggested action: ✅ Safe to merge — both reviewers found no blocking issues.

Reviewer A · gpt-oss-120b — recommends ✅ merge
The PR introduces a new telemetry system for tracking false‑positive gate blocks, adds a new DB table with migration, implements recording and resolution logic, exposes an internal API endpoint, and provides comprehensive unit and integration tests. The changes are well‑scoped, type‑safe, and respect the public/private boundary.

Suggestions

• Add a short comment in the migration file indicating it should be run during the next release to avoid missing the new table in production.
• Document the new internal endpoint in the API reference or README for maintainers.
• Consider increasing the row limit in listGateOutcomes or adding pagination for future scalability.

Worth double-checking

• Ensure the migration is applied in all environments; otherwise the new queries will fail.
• Verify that the internal auth middleware correctly protects the new route to prevent accidental public exposure.

Reviewer B · nemotron-3-120b-a12b — recommends ✅ merge
The PR adds internal telemetry for gate false-positive rates (#554) to help maintainers evaluate gate precision. It introduces a new service, database table, migration, types, internal API route, and webhook hooks to record hard blocks and correlate them with PR merges/overrides. Changes are measurement-only, respect the public/private boundary (internal-only endpoint, no forbidden terms), and include comprehensive unit and integration tests. The code follows existing patterns and properly handles edge cases like re-blocking and write failures.

No blocking issues spotted.

[codecov]

Codecov Report

❌ Patch coverage is 96.00000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.55%. Comparing base (07d7dae) to head (e0193aa).

Files with missing lines	Patch %	Lines
src/db/repositories.ts	92.85%	0 Missing and 1 partial ⚠️
src/db/schema.ts	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #755      +/-   ##
==========================================
- Coverage   96.55%   96.55%   -0.01%     
==========================================
  Files          97       98       +1     
  Lines       14091    14141      +50     
  Branches     5141     5151      +10     
==========================================
+ Hits        13606    13654      +48     
- Misses        105      106       +1     
- Partials      380      381       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JSONbored]

PR #755 — feat(gate): #554 false-positive telemetry for hard-blocked PRs (measurement-only)
Verdict: REQUEST CHANGES
Resolves issue #554: FULLY
CI: Prior codecov/patch failure is RESOLVED (not present; security/workers/changes green). lint/test/ui were still in_progress at review time — no failures observed.

Highlights:

• Well-isolated design: gate_outcomes table + schema, recordGateBlockOutcome/resolveGateOutcome helpers, pure buildGateFalsePositiveReport aggregator; processor hooks are best-effort try/catch so telemetry can never disrupt gate/webhook processing (asserted via throwing-DB test).
• Correct, tested false-positive semantics: blocked-then-merged -> FP; plain close unresolved (true positive); re-block clears prior resolution. Internal-auth-only endpoint with 401 negative test; aggregate-only report (no PII/reward/trust fields).

Concerns / required changes:

• Migration number collision: adds migrations/0037_gate_outcomes.sql while PR #753 also adds migrations/0037_badge_enabled.sql (main is at 0036). Whichever merges second must renumber to 0038 — coordinate ordering.
• "overridden" resolution exists in the type/aggregator (counted as FP) but no recording path ever writes it — the #538 override correlation is effectively a stub; acceptable as forward-compat but untested end-to-end.
• failing CI (codecov) - requires additional test coverage to meet threshold

Completeness vs issue: Fully implements the AC (blocked-then-merged increments FP count; rate queryable per gate mode) with privacy-safe aggregation and meaningful tests. The 0037 collision is the one required fix.

[JSONbored]

Thanks for this, @bittoby — genuinely solid work. 🙏

Your #755 and the maintainer PR #828 independently landed on the same issue (#554), and our own Gittensory Gate flagged the overlap as a duplicate on both PRs — a nice bit of dogfooding for the very feature this issue adds (gate-outcome telemetry).

We're going with #828 because it was already green and is wired into a couple of things that landed in parallel: the #538 maintainer-override path (an override becomes the strongest false-positive signal), the config-as-code surface, and it mirrors the existing #543 outcome-learning loop's ledger + service split. Your structure was very close to the same shape — the gate_outcomes table, the recording site, the aggregation service, the tests. The only mechanical gaps were CI (lint/validate + the 97% patch-coverage gate) and the migration number (0040 got taken by another PR mid-flight — exactly the kind of cross-PR migration collision we have a guard for now).

Closing this in favor of #828, but please don't let that discourage you — this was a real, well-scoped contribution and we'd love more. If you want a good next one, #540 (advisory reviewer routing via CODEOWNERS) is open and unclaimed. Thank you again. 🚀

[Crystora]

@JSONbored Hi, why you closed my pr? I am updating pr according to your feedback currently.

[Crystora]

did you see my commits? I just committed them about 2 mins ago before you closed my pr.

[Crystora]

I suspect it was intentionally done to close my pr.