Skip to content

24 console.error calls in the queue pipeline carry a mismatched level: "warn", downgrading real errors to Sentry warnings #7806

Description

@JSONbored

Context

src/selfhost/sentry.ts's forwardStructuredLogToSentry decides a log line's Sentry severity from the structured JSON payload's own level field when present, and only falls back to a sink-based default (console.error"error") when level is absent (src/selfhost/sentry.ts:641-646):

// A console.error sink is error-level by DEFAULT even when the JSON omits an explicit level (many engine error
// logs do) -- that's how those errors reach Sentry instead of printing to stderr and vanishing. An EXPLICIT level
// ...
const explicitLevel = typeof obj.level === "string" ? obj.level : undefined;
const level = explicitLevel ?? (fromErrorSink ? "error" : undefined);

An explicit level in the payload always wins over the console.error sink default. In src/queue/processors.ts, src/queue/slop-detection.ts, and src/queue/ai-review-orchestration.ts, 24 call sites call console.error(JSON.stringify({ level: "warn", ... })) — using the console.error sink (correctly signaling a real failure to stderr) but explicitly stamping the payload with level: "warn", which downgrades these to warning severity in Sentry per the logic above. Two representative examples:

// src/queue/processors.ts:9060-9062
await markPullRequestSurfacePublished(env, repoFullName, pr.number, advisory.headSha).catch((error) => {
  console.error(JSON.stringify({ level: "warn", event: "surface_published_mark_failed", repoFullName, pullNumber: pr.number, error: errorMessage(error) }));
});
// src/queue/slop-detection.ts:172-182
} catch (error) {
  console.error(
    JSON.stringify({
      level: "warn",
      event: "ai_slop_failed",
      ...
    }),
  );
}

The same three files correctly pair console.error with level: "error" (or omit level and rely on the documented sink default) at 46 other call sites — e.g. src/queue/processors.ts:1112, :3828, :5576, :6389's neighbors, and many more. The 24 mismatched sites are the exception, not the convention, and are load-bearing for observability: every one of them is a .catch()/catch handler for a genuine DB-write, sweep, or AI-review failure (gate_check_summary_upsert_failed, ai_review_failed, ai_slop_failed, sweep_mark_regated_failed, decision_pack_login_failed, and 19 others), so this silently downgrades real production errors to warning-level in Sentry across the queue pipeline — undermining error-rate alerting for exactly the failure paths that matter most (best-effort DB writes and AI-review/slop advisory calls that already swallow their own errors and rely on Sentry as the only remaining visibility).

Confirmed mechanically (no other console-method/level mismatches exist anywhere else in src/review/, src/queue/, or src/signals/console.warn calls in these directories all correctly pair with level: "warn"/"warning").

Requirements

  • Every console.error(JSON.stringify({ level: "warn", ... })) call site listed below must have its level field changed to "error", matching the convention already used by the other 46 console.error call sites in the same three files.
  • No other field, event name, or log content changes.
  • Exact call sites (line numbers as of this issue's filing — re-verify against current HEAD before editing, since intervening PRs may shift them):
    • src/queue/slop-detection.ts:173
    • src/queue/ai-review-orchestration.ts:881
    • src/queue/processors.ts:806, 1475, 1776, 1920, 3585, 3619, 5140, 6389, 6427, 6458, 6484, 6509, 6613, 7014, 7480, 7558, 9061, 9066, 10375, 10467, 10513, 10555

Deliverables

  • All 24 call sites listed above changed from level: "warn" to level: "error".
  • A regression test (in whichever existing test file already exercises one or more of these failure paths, e.g. a DB-write-failure or AI-review-failure test) asserting the logged JSON payload's level field is "error" for at least one representative call site, so a future edit can't silently reintroduce the mismatch.

Test Coverage Requirements

src/queue/** is under the top-level 99% patch coverage gate — every changed line (all 24 call sites) must be covered; the codebase's existing failure-path tests likely already exercise most of these .catch()/catch branches, so this is primarily a targeted-assertion addition rather than new branch coverage.

Expected Outcome

Every console.error call in src/queue/processors.ts, src/queue/slop-detection.ts, and src/queue/ai-review-orchestration.ts carries a level matching the sink it uses ("error"), so forwardStructuredLogToSentry classifies these 24 genuine failure conditions as Sentry errors instead of warnings, consistent with every other console.error call site in the same files.

Links & Resources

  • src/selfhost/sentry.ts:641-646 (the severity-resolution logic: an explicit level always wins over the sink default) and :777-779 (documents the intended console.error → error-by-default contract)
  • src/queue/processors.ts:1112 (an example of the correct, already-existing level: "error" convention in the same file)
  • The 24 call sites listed in Requirements above

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions