obs(selfhost): detect and alert on GitHub App JWT clock skew

[JSONbored]

Summary

• Fixes obs(selfhost): detect and alert on NTP/system clock drift #3811. edge-us-01's system clock silently drifted ~3 minutes off true time because its sole configured NTP source was dead (chronyc sources showed Reach: 0 the whole time, no redundant fallback), breaking GitHub App JWT auth ("Bad credentials") for a window before anyone noticed.
• Adds gittensory_clock_skew_seconds, a gauge sampled from the Date response header of the GitHub App's own JWT-authenticated installation-token mint call (requestInstallationTokenWithJwt, src/github/app.ts) — no extra network round-trip, sampled at exactly the cadence the vulnerable code path itself runs. State lives in the new src/selfhost/clock-skew.ts, registered as a gauge in src/server.ts alongside the other self-host-only host gauges.
• Adds prometheus/rules/alerts.yml warning (60s) and critical (120s) drift alerts — both comfortably under the 3-minute drift actually observed, and tied directly to GitHub App JWTs' 60s iat backdate margin (createAppJwt).
• Adds a Clock Sync (NTP, #3811) row + Clock Skew panel to grafana/dashboards/gittensory.json.
• Adds an NTP guidance section to the self-host operations docs (docs.self-hosting-operations.tsx) and a preamble comment in docker-compose.yml pointing operators at configuring redundant host NTP sources — a single dead source is a silent single point of failure.

Not included: correlating a JWT 401 rejection with elevated clock skew via captureError (a "consider" item in the issue, not a hard requirement). captureError/Sentry is explicitly self-host-only tooling (only ever called from src/server.ts/src/selfhost/*-queue.ts), and src/github/app.ts is shared cloud+self-host code with no existing precedent for importing it — the Prometheus alert rules are the primary, always-on alerting mechanism this issue asks for, so I kept the JWT retry path (already delicate, incident-driven code per #2453) untouched rather than adding a new cross-boundary import for a soft suggestion.

Scope

• The PR title follows type(scope): short summary Conventional Commit format, for example fix(api): restore profile access checks.
• This PR is focused and does not mix unrelated backend, UI, MCP, docs, dependency, and deploy changes.
• This follows CONTRIBUTING.md and does not reintroduce GitHub Pages, VitePress, site/, or CNAME.
• I linked an issue, or this is small enough that the summary explains why an issue is not needed.

Validation

• git diff --check
• npm run actionlint
• npm run typecheck
• npm run test:coverage locally; codecov/patch requires ≥99% coverage of the lines AND branches you changed (aim for 100% on your diff so CI variance does not fail near the threshold). Global coverage is a non-blocking trend with a loose 90% backstop, not the gate.
• npm run test:workers
• npm run build:mcp
• npm run test:mcp-pack
• npm run ui:openapi:check
• npm run ui:lint
• npm run ui:typecheck
• npm run ui:build
• npm audit --audit-level=moderate
• New or changed behavior has unit/integration tests for new branches, fallback paths, and sanitizer boundaries

All run via npm run test:ci (full local gate, green) plus a standalone npm audit --audit-level=moderate (0 vulnerabilities). Also ran npm run selfhost:validate-observability (validates the new Prometheus rule YAML and Grafana dashboard JSON), docker compose config --quiet (validates the compose comment doesn't break parsing), and npm run docs:drift-check / npm run selfhost:env-reference:check (no new env flags introduced — sampling is unconditional). Verified every new/changed line and branch in src/selfhost/clock-skew.ts and src/github/app.ts's changed section is covered locally (v8 lcov branch report).

Safety

• No secrets, wallet details, hotkeys, coldkeys, user PATs, private keys, raw trust scores, private rankings, or private maintainer evidence are exposed.
• Public GitHub text stays sanitized, low-noise, and does not imply compensation guarantees or optimization tactics.
• Auth, cookie, CORS, GitHub App, Cloudflare, or session changes include negative-path tests.
• API/OpenAPI/MCP behavior is updated and tested where needed.
• UI changes use live API data or real empty/error/loading states, not production mock/demo fallbacks.
• Visible UI changes include a UI Evidence section below with JPG/JPEG or PNG screenshots arranged as organized, captioned, clickable thumbnails.
• Public docs/changelogs are updated where needed; changelogs are only edited for release-prep PRs.

This touches the GitHub App JWT-mint response path but only ADDS a passive header read (recordClockSkewFromResponse) after the response is already received — no change to auth logic, retry behavior, or token caching, so no new auth-surface negative-path test is needed. The self-hosting-operations doc addition is a docs/text change (not an interactive UI surface), so no screenshot evidence applies.

UI Evidence

N/A — this PR only adds a docs section (text/prose) and a Grafana JSON panel definition; no interactive app UI surface changed.

Notes

• Deferred (see Summary): correlating a JWT 401 with elevated clock skew via captureError. Left as a possible follow-up once/if a self-host-only wrapper around the App-JWT retry path exists.

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	gittensory-ui	6de2833	Commit Preview URL Branch Preview URL	Jul 06 2026, 07:43 PM

[superagent-security]

Superagent didn't find any vulnerabilities or security issues in this PR.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.48%. Comparing base (a873a46) to head (6de2833).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3861   +/-   ##
=======================================
  Coverage   93.48%   93.48%           
=======================================
  Files         333      334    +1     
  Lines       33162    33172   +10     
  Branches    12128    12130    +2     
=======================================
+ Hits        31001    31011   +10     
  Misses       1530     1530           
  Partials      631      631           

Files with missing lines	Coverage Δ
src/github/app.ts	98.18% <100.00%> (+0.01%)	⬆️
src/selfhost/clock-skew.ts	100.00% <100.00%> (ø)
src/selfhost/metrics.ts	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[loopover-orb]

Warning

🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨🟨

⏸️ Gittensory review result - manual review recommended

_{Review updated: 2026-07-06 19:54:36 UTC}

10 files · 1 AI reviewer · no blockers · readiness 100/100 · CI green · clean

⏸️ Suggested Action - Manual Review

• Touches a guarded path — held for manual review: This PR changes guardrail-protected path(s): src/selfhost/clock-skew.ts (matched src/selfhost/**), src/selfhost/metrics.ts (matched src/selfhost/**).

Review summary
This adds a `gittensory_clock_skew_seconds` gauge sampled opportunistically from the `Date` header of the existing JWT-authenticated installation-token mint call in `src/github/app.ts`, wires it into `src/server.ts`/`metrics.ts`, adds Prometheus warning/critical alerts, a Grafana panel, and operator docs — a low-risk, well-targeted observability addition with no new network calls. The `recordClockSkewFromResponse` logic in `src/selfhost/clock-skew.ts` correctly ignores missing/unparseable `Date` headers rather than resetting to 0, and the accompanying unit tests exercise both skew directions and the ignore-on-bad-header paths using fake timers. The one gap is that the gauge only updates when a token mint actually occurs — if minting is infrequent (long-lived cached tokens), the metric can go stale silently, which the PR description itself acknowledges is a deliberate tradeoff to avoid a new network round-trip.

Nits — 7 non-blocking

• src/selfhost/clock-skew.ts: the module-level `lastSkewSeconds` never expires — if installation-token minting is infrequent (long cache lifetime), a stale sample sits at whatever value was last observed with no staleness indicator on the metric itself; consider exposing a last-sampled timestamp or making that staleness explicit in the docs.
• prometheus/rules/alerts.yml: both new alerts key off the same single gauge with no `by(...)` label if the self-host deployment ever runs multiple processes/hosts — worth confirming this metric is per-instance-scraped rather than aggregated across hosts, since clock skew is host-specific.
• test/unit/github-app.test.ts:207 — the review brief flagged a 'generic_secret_assignment' at this line; it's almost certainly the test's generated/stubbed private key (consistent with `generatePrivateKeyPem()` used elsewhere in the file), not a real credential, but worth a quick human confirmation.
• docker-compose.yml comment addition is documentation-only and correctly notes containers share the host clock — no functional change to verify there.
• Consider recording a `lastSampledAtMs` alongside `lastSkewSeconds` in clock-skew.ts and exposing a companion `gittensory_clock_skew_sample_age_seconds` gauge so operators can distinguish 'no drift' from 'no recent sample' on the Grafana panel.
• PR author also opened the linked issue — Link an issue that was opened by a different contributor, or provide a rationale for why this self-authored issue represents genuine discovery work.
• Touches a guarded path — held for manual review — A maintainer must review and merge this change.

Signal	Result	Evidence
Code review	✅ No blockers	1 reviewer
Linked issue	✅ Linked	#3811
Related work	✅ No active overlap found	No same-issue or scoped active PR overlap found.
Change scope	✅ 20/20	Low review scope from cached public metadata (1 linked issue).
Validation posture	✅ 25/25	PR body includes validation/test evidence.
Contributor workload	✅ 10/10	Author activity: 51 registered-repo PR(s), 43 merged, 382 issue(s).
Contributor context	✅ Confirmed Gittensor contributor	JSONbored; Gittensor profile; 51 PR(s), 382 issue(s).
Gate result	⚠️ Not blocking	Advisory; not blocking this PR.

Review context

• Author: JSONbored
• Role context: owner (maintainer lane)
• Public audience mode: oss maintainer
• Lane context: Repository is configured for direct PR review.
• Public profile languages: Python, TypeScript, JavaScript, Ruby, Go, Kotlin, MDX, Shell
• Official Gittensor activity: 51 PR(s), 382 issue(s).
• PR-specific overlap: none found.

Contributor next steps

• Treat this as maintainer-lane context rather than normal contributor-lane activity.

Signal definitions

• Related work = same linked issue, overlapping active PRs, or title/path similarity.
• Change scope = cached public metadata such as size labels, draft state, and review-burden hints.
• Validation posture = whether the PR provides enough public validation/test evidence for maintainer review.
• Contributor workload = public contributor activity and cleanup pressure, not a repo-wide quality failure.
• Contributor context = public GitHub/Gittensor identity context; non-Gittensor status is not a blocker.

_{🟩 Safe / merged · 🟦 Advisory · 🟨 Held for review · 🟥 Blocked / closed}

---

💰 Earn for open-source contributions like this. Gittensor lets GitHub contributors earn for the work they already do — register to start earning →.

Checked by Gittensory, a quiet PR intelligence layer for OSS maintainers.

• Re-run Gittensory review