Skip to content

feat(selfhost): observability hooks — @sentry/node error tracking module + boot/queue/review capture seams #1468

Description

@JSONbored

Context

The self-host Node engine has no error tracker. Crashes at boot, terminal dead-letter job failures, and degraded/timed-out claude --print reviews are only visible as structured console.error lines to Loki — there is no aggregation, fingerprinting, or alerting. Add opt-in @sentry/node error tracking that is a complete no-op when SENTRY_DSN is unset, mirroring the existing env-gated, dynamically-imported selfhost-integration pattern (Redis/Qdrant/embed-provider in src/server.ts). The Worker entry src/index.ts and the Worker bundle stay untouched.

This is the code-wiring PR; source-maps + docs + Grafana link ship in the sibling issue.

Requirements

  • Add @sentry/node to package.json dependencies; commit the lockfile. It must be referenced ONLY from src/server.ts and the new src/selfhost/sentry.ts — never imported into src/index.ts, so cloudflare:* stubbing stays clean.
  • No top-level @sentry/node import in sentry.ts; lazy await import(...) inside initSentry only, so the package never enters the Worker bundle.
  • Use an explicit NodeClient + Scope (NOT the global Sentry.init()), so we do NOT install Sentry's default uncaughtException handler — we own those handlers to coexist with the existing main().catch(...) fatal catch and the graceful-shutdown signal handlers (src/server.ts:423-433).
  • beforeSend secret-scrubber: drop request.headers.authorization/cookie/x-hub-signature*; redact any value matching token/key/secret/gh[pousr]_/ANTHROPIC/sk- patterns in extra/contexts; strip event.server_name if it would leak a private host. No webhook secret or short-lived Orb token may ride along in a crash report.
  • tracesSampleRate defaults to 0 (errors-only).

Deliverables

  • src/selfhost/sentry.ts:
    • initSentry(env): Promise<boolean> — returns false (no import, no client) when SENTRY_DSN is unset; otherwise dynamic-imports @sentry/node, builds a NodeClient with dsn/environment(SENTRY_ENVIRONMENT??'production')/release(SENTRY_RELEASE??GITTENSORY_VERSION)/serverName(PUBLIC_API_ORIGIN)/tracesSampleRate(SENTRY_TRACES_SAMPLE_RATE??0) + beforeSend: scrubEvent, attaches it to an explicit Scope, and client.init().
    • captureError(err, ctx?) and captureReviewFailure(err, ctx) (the latter level:'warning', sets owner/repo/pr/head_sha as Sentry TAGS not just context) — both unconditional no-ops when the client is undefined, so call sites need no if (sentryOn) guard.
    • flushSentry(timeoutMs=2000) — no-op when off; .catch(() => undefined).
  • src/server.ts: call initSentry(process.env) right after loadFileSecrets() (so SENTRY_DSN_FILE Docker-secret resolution has run) in main(); add process.on('uncaughtException'|'unhandledRejection') handlers that capture+flush(+exit on uncaught); convert main().catch(...) to capture+flush before process.exit(1); add captureError to the cron-tick catch (selfhost_cron_error) and orb-export catch (selfhost_orb_export_error); add await flushSentry() inside shutdown() after server.close(), before backend.shutdown().
  • src/selfhost/sqlite-queue.ts: capture ONLY on terminal dead-letter (not transient retries) at the unparseable-payload branch and the max-retries-exhausted branch, fingerprinted by extractPayloadType(job.payload) (already imported). Add the identical one-liner to src/selfhost/pg-queue.ts's dead-letter seam for parity.
  • src/queue/processors.ts review case: wrap the review invocation so an UNEXPECTED throw (e.g. a GitHub 403 mid-review, not the never-throw AI path) is captured with {kind:'review',owner,repo,pr,head_sha,delivery_id} before re-throwing into retry/dead-letter. src/services/ai-review.ts: add captureReviewFailure(...) inside the existing never-throw catches (the BYOK/CLI catch + advisory catch) so degraded/timed-out subprocess reviews are visible WITHOUT changing the swallow-and-degrade behavior the gate depends on.

Acceptance criteria

  • SENTRY_DSN unset: initSentry returns false, @sentry/node is never imported, and captureError/captureReviewFailure/flushSentry are safe no-ops — byte-identical engine behavior to today.
  • src/index.ts and the Worker bundle are unchanged; build of the Worker still stubs cloudflare:* cleanly.
  • Tests under test/unit/ cover: no-op-when-unset; beforeSend redacts secrets/headers/server_name; dead-letter capture invoked exactly once on terminal failure and NOT on transient retry; review-failure capture invoked with repo/PR/SHA tags. Both arms of every nullish/conditional covered to clear 97% branch-counted patch on src/**.
  • npm run test:coverage (unsharded) at 100% patch on changed src/**; npm run test:ci green; npm audit --audit-level=moderate clean. No schema/migration change expected.

Parent: #1029

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions