Skip to content

ci: shorten the feedback loop (parallel jobs, warm Go cache, Build Cloud) - #4220

Merged
dgageot merged 4 commits into
mainfrom
ci-improvements
Sep 10, 2026
Merged

dgageot merged 4 commits into
mainfrom
ci-improvements

Conversation

@aheritier

@aheritier aheritier commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Why

A main run took ~13–15 min and a PR run ~8–10 min. Per-step timings showed the time goes to compiling, not testing: every job recompiled the whole dependency tree because the Go build cache was effectively cold (see commit 2), and the image pipeline used 2× the runners it needed.

What (one commit each)

Runners are left as-is: on a public repo ubuntu-latest/windows-latest are already 4-core, and the org's larger runners are out of scope here.

  1. Parallelize compile-only checks and image builds — check-plan-cross + test-wasm-providers move to a cross-compile-checks job; build-and-push-image no longer waits on tests (it pushes untagged digests only), the gate moves to the tag-publishing job.
  2. Per-job warm Go cache — setup-go's built-in cache is keyed on go.sum alone, so one snapshot per OS was saved on 2026-08-31 and every job since logged Cache hit … not saving cache. New composite .github/actions/setup-go restores GOMODCACHE+GOCACHE keyed <os>-<job>-<go.sum>-<sha> with prefix fallback; saved on main only. Also drops the redundant go test ./e2e -run TestWasmProviderDependencies from CI (already covered by task test; cold it cost ~2 min to compile e2e).
  3. One Build Cloud job per stage — Build Cloud compiles both platforms natively, so the per-platform matrices only doubled runners. Digests flow as job outputs, no artifacts; the two merge jobs collapse into publish-tags. cache-binary: false on buildx stops a ~40 MB cache write per PR branch.

Measured so far

PR run 34463832719 vs the last run on the previous layout (34458544304). Caches are not seeded yet (saves happen on main only), so every job here starts cold — this is the floor, not the target.

job before this PR note
required checks green 8:15 ~5:45
lint 3:08 4:10 cold module download until main seeds a cache
build-and-test 8:15 5:44 compile-only checks moved out
cross-compile-checks (in build-and-test) 4:31 e2e trim; cold
windows-tests 7:52–9:49 10:36 Set up Go 2:19 → 0:23 (no stale 944 MB restore); Run tests 5:46 → 9:58 cold
license-check 1:01 1:19 cold
build-image 2 runners 1 runner, 1:07 both platforms in one Build Cloud job

After merge: the first main run seeds one cache per job; the next PR run is the real measurement (expected: build ~20 s, lint <1 min, cross-compile <1 min, Windows tests ~3–4 min). Windows cache size is the thing to check — if it lands well above 1 GB the restore eats into the gain.

Tradeoffs

  • A failed test run leaves untagged digests on Hub (nothing user-visible moves; GC'd eventually).
  • Each main run saves 5 caches (~0.5–1 GB each). The repo is at the 10 GB cap (503 caches, mostly per-PR buildx-dl-bin-*), so stale entries should be purged once this lands.
  • Windows cache restore is slow (2+ min for ~1 GB); net gain expected but to be confirmed.

Follow-ups (repo settings, not in this PR)

  • Add cross-compile-checks and windows-tests to the main ruleset's required checks (currently only build-and-test, license-check, lint).
  • Purge stale caches: gh cache list --limit 500 --json id,key --jq '.[] | select(.key|test("^(buildx-dl-bin|setup-go-)")) | .id' | xargs -n1 gh cache delete

🤖 Generated with Claude Code

Enable Docker build record upload and summary in the CI workflow
to capture build metadata and provide better visibility into the
build process. This allows tracking of build performance metrics
and build history in Docker Cloud.
…d image builds

build-and-test spent ~3 of its ~9.5 minutes on check-plan-cross and
test-wasm-providers, neither of which depends on the test run. Move them
to a cross-compile-checks job that runs alongside it.

build-and-push-image waited on all four test jobs before starting even
though it only pushes untagged per-arch digests. Let it run in parallel
and put the lint/test gate on the merge-and-push-* jobs, which are the
only ones that move tags.

Together this cuts the main pipeline from ~13 min to ~8 min.
@aheritier aheritier added the area/ci CI/CD workflows and pipeline label Sep 10, 2026
setup-go's built-in cache is keyed on go.sum alone, so one snapshot per
OS was saved on 2026-08-31 and every job since restored that stale copy
with "cache hit, not saving". Every job recompiled the whole dependency
tree: build 2:09, golangci-lint 2:24, cross-compile 3:01, Windows tests
6:57.

Replace it with a composite setup-go action that restores a per-job
GOMODCACHE + GOCACHE keyed on <os>-<job>-<go.sum>-<sha> with prefix
fallback, and save it at the end of each job on main only. PR runs
restore from the latest main cache and never write, so cache churn stays
bounded.

Also stop running `go test ./e2e -run TestWasmProviderDependencies` in
cross-compile-checks: `task test` already covers it via ./..., and cold
it cost ~2 min just to compile the e2e package. Split the GOOS=js half
into a test-wasm-js task for CI; test-wasm-providers keeps both for
local use.
Build Cloud compiles amd64 and arm64 natively regardless of the GitHub
runner's architecture, so the per-platform matrices only doubled runner
count and setup overhead. Build both platforms in a single job per
stage: build-image on PRs, build-and-push-image on main/tags.

Pushing a multi-platform build by digest yields one untagged index, so
the digest travels as a job output instead of per-arch artifacts, and
the two merge jobs collapse into publish-tags, which still carries the
lint/test gate and is the only job that moves tags.

Disable setup-buildx's binary cache: the download takes seconds and the
cache wrote a ~40 MB entry per PR branch, crowding the 10 GB repo limit.

Net: 4 fewer runners per main run, 2 fewer per PR run, no artifacts.
@aheritier aheritier changed the title ci: parallelize compile-only checks and image builds ci: shorten the feedback loop (parallel jobs, warm Go cache, Build Cloud, 8-core runners) Sep 10, 2026
@aheritier aheritier changed the title ci: shorten the feedback loop (parallel jobs, warm Go cache, Build Cloud, 8-core runners) ci: shorten the feedback loop (parallel jobs, warm Go cache, Build Cloud) Sep 10, 2026
@aheritier aheritier added the kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix) label Sep 10, 2026
@aheritier
aheritier marked this pull request as ready for review September 10, 2026 10:35
@aheritier
aheritier requested a review from a team as a code owner September 10, 2026 10:35
@dgageot
dgageot merged commit 070f654 into main Sep 10, 2026
17 checks passed
@dgageot
dgageot deleted the ci-improvements branch September 10, 2026 11:07
aheritier added a commit that referenced this pull request Sep 10, 2026
Fork PRs receive neither repository variables nor an OIDC token, so the
image job failed on `DOCKERHUB_OIDC_CONNECTION_ID must be configured` for
the 60% of PRs that come from forks (the reason #4220 was reverted in #4223).
The old condition compared the PR's *base* repository to this one, which is
always true; trust is now decided on the *head* repository.

Trusted runs keep Build Cloud and both platforms. Fork PRs build linux/amd64
on the runner with a plain docker-container builder, no login, nothing
pushed, no build-record upload (the read-only token cannot create artifacts).
The Dockerfile is exercised for every PR instead of being skipped.
aheritier added a commit that referenced this pull request Sep 10, 2026
…ally

Fork PRs, PRs opened by Dependabot and other bots running with a read-only
token, and CI in forks of this repo receive neither repository variables nor
an OIDC token, so the image job failed on `DOCKERHUB_OIDC_CONNECTION_ID must
be configured` (why #4220 was reverted in #4223). The old condition compared
the PR's *base* repository to this one, which is always true.

The job now decides at runtime: Build Cloud (both platforms, one runner)
whenever the Hub OIDC connection id and the OIDC token endpoint are present,
otherwise a plain docker-container builder on the runner, linux/amd64 only,
nothing pushed, no build-record upload. Publishing runs still fail loudly
when Build Cloud is unavailable rather than downgrading. The Dockerfile is
exercised on every PR instead of being skipped or failing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci CI/CD workflows and pipeline kind/chore Maintenance, deps, CI, tooling (maps to chore: commit prefix)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants