CI_FAILING_CONCLUSIONS includes "cancelled" (src/github/backfill.ts ~2581), and the dedupe key is name \\0 appSlug \\0 checkSuiteId (~2848-2852) — deliberately not collapsed across suites, so a failure cannot be hidden by a later success.
That protection has a bad interaction. A same-SHA workflow re-trigger (reopened, ready_for_review, labeled — ORB itself applies labels) creates a new check suite. A workflow with concurrency: cancel-in-progress: true then cancels the first run, leaving conclusion: "cancelled" check-runs attached to the same head SHA in the old suite.
Those cancelled runs land in failingDetails alongside the new suite's passing runs → ciState = "failed" → willClose fires on ciFailed (src/settings/agent-actions.ts ~1074) → one-shot auto-close of a PR whose CI is green. The executor's recheck re-derives the same "failed", so it confirms rather than catches it.
This is especially concerning because CI is failing (...) is the single most common close reason in our ledger, so any false-positive rate here is expensive — and ORB applying a label is itself a trigger for the re-run that creates the second suite.
Status: SUSPECTED — needs one empirical check. On the live box, find a PR head SHA with ≥2 check suites and GET /commits/{sha}/check-runs?filter=latest; if any returned run has conclusion: "cancelled" with a check_suite.id older than the newest suite, this is CONFIRMED.
Fix
Ignore a cancelled run whose suite is superseded by a newer suite carrying the same check name, or treat cancelled as non-terminal (pending) rather than failing. Keep the cross-suite non-collapse for genuine failure conclusions.
Acceptance
- A PR whose first suite was concurrency-cancelled and whose second suite is green resolves to
passed.
CI_FAILING_CONCLUSIONSincludes"cancelled"(src/github/backfill.ts~2581), and the dedupe key isname \\0 appSlug \\0 checkSuiteId(~2848-2852) — deliberately not collapsed across suites, so a failure cannot be hidden by a later success.That protection has a bad interaction. A same-SHA workflow re-trigger (
reopened,ready_for_review,labeled— ORB itself applies labels) creates a new check suite. A workflow withconcurrency: cancel-in-progress: truethen cancels the first run, leavingconclusion: "cancelled"check-runs attached to the same head SHA in the old suite.Those cancelled runs land in
failingDetailsalongside the new suite's passing runs →ciState = "failed"→willClosefires onciFailed(src/settings/agent-actions.ts~1074) → one-shot auto-close of a PR whose CI is green. The executor's recheck re-derives the same"failed", so it confirms rather than catches it.This is especially concerning because
CI is failing (...)is the single most common close reason in our ledger, so any false-positive rate here is expensive — and ORB applying a label is itself a trigger for the re-run that creates the second suite.Status: SUSPECTED — needs one empirical check. On the live box, find a PR head SHA with ≥2 check suites and
GET /commits/{sha}/check-runs?filter=latest; if any returned run hasconclusion: "cancelled"with acheck_suite.idolder than the newest suite, this is CONFIRMED.Fix
Ignore a
cancelledrun whose suite is superseded by a newer suite carrying the same check name, or treatcancelledas non-terminal (pending) rather than failing. Keep the cross-suite non-collapse for genuinefailureconclusions.Acceptance
passed.