Repository navigation
Run only the CI a pull request's change can affect, and everything nightly - #882
Conversation
Sensitive Change Detection (shadow mode)This PR modifies control-plane files:
|
There was a problem hiding this comment.
🟡 Changes recommended
The shared non-PR concurrency group can cancel pending main, scheduled, or dispatched runs.
1 open finding
What changed in this PR
Adds change-aware CI filtering for pull requests while preserving full scheduled and manually triggered checks.
Changes:
- Classifies changes to selectively run Go, skill, and documentation checks.
- Adds nightly execution and cancellation of superseded PR runs.
- Keeps required checks reportable through job-level conditions.
[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or rungh pr ready --undo.
Click "Ready for review" or rungh pr readyto reengage.
| File | Description |
|---|---|
.github/workflows/test.yml |
Adds scheduling, concurrency, change classification, and conditional CI execution. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 1 file
You’re at about 92% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
|
🤖 Measured. Each kind of change, from the Test workflow's jobs (wall clock is first job start to last job end):
Every required check name reported on every run: none stuck pending. The skipped ones are skipped check runs under the exact required names, from GitHub Actions. GitHub counts those as passing for a required check (docs: "It will not prevent a pull request from merging, even if it is a required check"). One limit on that proof: the probes targeted this branch, not Runner-minutes for the code-change rows are inflated by the queueing while all these PRs ran at once. Wall clock is the comparable number. |
…ghtly Every pull request ran the whole Go pipeline, about 6 minutes and 32 runner-minutes, even when it changed only prose or a skill's text. basecamp-cli#878 (29 lines of SKILL.md and an eval case) ran all fourteen Test jobs. A new Classify changes job sorts each pull request's files: - Anything outside two short lists runs everything, as before. - Skill and eval files only (skills/**/*.md, skill-evals/**) run Tests and Integration Tests, which read the embedded skill, its drift fixtures and the eval patterns. Lint, Security, CLI Surface Check and the race shards skip: none of them can see that change. - Prose only (root *.md, .github/*.md) runs only Tests' naming check. Fail-safe in both directions. A path narrows the run only if it is listed, so an unlisted file and any change to this workflow run everything. A classifier that fails or answers nothing narrows nothing: the gates read != 'false' under !cancelled(), so an empty answer runs the job. Required checks keep reporting. The gates are job-level ifs, and GitHub counts a job skipped by its condition as a passing check. A workflow-level paths filter would leave Tests, Lint, Security, Race Detection and Integration Tests pending forever. Race Detection keeps always(), so a failed shard still fails it, and it skips only when the shards skipped by design. Pushes to main, a new nightly schedule and dispatches are never narrowed. The schedule also gives govulncheck, which lives only in this workflow, a daily run against the moving vulnerability database: the GO-2026-6617 advisory sat unseen on main until someone pushed. A new push to a pull request now cancels that pull request's run in flight. Main, the schedule and dispatches never cancel.
A concurrency group keeps one running and one pending run, and a newer pending run replaces the older one even with cancel-in-progress off. With every main push, nightly run and dispatch in test-refs/heads/main, a push could silently drop a queued nightly run. Pull requests keep one group per PR, so a new push still supersedes the run in flight.
The job and step names already say what each one does. The reasoning for why it's safe (fail-safe lists, job-level skips that report as passing, Race Detection's always()) is in the pull request and the first commit's message.
b3b2b5a to
33dda43
Compare
Stacked on #881 ("Build with Go 1.26.9 for eleven standard-library advisories"), which is stacked on #880 ("Bump golang.org/x/net to v0.60.0 for GO-2026-6617"). It's stacked only so its own Security check runs green; the change here is just
.github/workflows/test.yml. When #880 merges, GitHub retargets this PR tomain.Problem
Every pull request runs the whole Go pipeline, whatever it changes. #878 ("Tell agents that rich-text fields support Mermaid diagrams") changed 29 lines of
SKILL.mdand one eval case, and still ran all 14 Test jobs. Measured on its first commit:The last 40 Test runs took 4.6–7.0 min, most of them 5.7–6.2. The race shards set the wall clock.
Change
A new Classify changes job sorts a pull request's files with
dorny/paths-filter(already pinned in this workflow,predicate-quantifier: every):skills/**/*.md,skill-evals/***.md,.github/*.mdcheck-namingscans Markdown)Pushes to
main,workflow_dispatchand a new nightly schedule (05:23 UTC) are never narrowed. The schedule also gives govulncheck a daily run against the moving vulnerability database. govulncheck lives only in this workflow, which is how GO-2026-6617 sat unseen onmainuntil someone pushed.Concurrency: a new push to a pull request cancels that PR's run in flight. Pushes to
main, the schedule and dispatches never cancel.Caching is already there.
setup-gov7 restores the module and build cache on every job (Cache hit for: setup-go-Linux-x64-ubuntu24-go-1.26.7-…in #878's Tests log). Nothing to add.Required checks still report
mainrequires Tests, Lint, Security, Race Detection, Integration Tests, Nix flake builds and Secret Scanning.paths. A filtered-out workflow never runs, and its required checks stay pending forever.if. GitHub reports a job skipped by its condition as passing: "A job that is skipped will report its status as 'Success'. It will not prevent a pull request from merging, even if it is a required check." (docs)always(). A failed shard still fails it. It skips only whengoisfalse, which is when the shards skipped by design.test.yml, including this PR. The gates read!cancelled() && needs.changes.outputs.X != 'false', so a failed or empty classifier runs every job. The classifier's own step rejects any answer other thantrueorfalse.Proof
test.yml, so its own run is the code path: everything runs.Timings (estimated now, measured on the probe below)
Not done here (proposals)
paths-ignorefor prose. It's left alone because code scanning flags PRs that have no analysis.Summary by cubic
Makes the test workflow change-aware: a pull request now runs only the jobs its changed files can affect, and a nightly schedule (05:23 UTC) runs the full suite. Previously every PR ran all 14 Test jobs (~6 min, ~32 runner-minutes) even when it changed only prose or a skill's text.
A new
Classify changesjob sorts each PR's changed files and is fail-safe both ways: any unlisted file, any change to this workflow, or a classifier that fails or answers nothing runs everything. The gates are job-levelifs under!cancelled(), so GitHub reports skipped required checks as passing instead of leaving them pending, and Tests never skips entirely (it still runs the naming check on prose-only PRs). Skill/eval-only PRs skip Lint, Security, CLI Surface Check and the race shards; prose-only PRs run only the naming check.Pushes to
main, dispatches and the nightly schedule are never narrowed. The schedule also gives govulncheck a daily run against the moving vulnerability database; it lives only in this workflow, which is how GO-2026-6617 sat unseen onmainuntil someone pushed.Concurrency: A new push to a pull request cancels that PR's run in flight. Every non-PR run gets its own concurrency group so a newer pending run can't silently replace a queued nightly run.
Written for commit 33dda43. Summary will update on new commits.