Repository navigation
ci: shorten the feedback loop (parallel jobs, warm Go cache, Build Cloud) - #4220
Merged
Merged
Conversation
Enable Docker build record upload and summary in the CI workflow to capture build metadata and provide better visibility into the build process. This allows tracking of build performance metrics and build history in Docker Cloud.
…d image builds build-and-test spent ~3 of its ~9.5 minutes on check-plan-cross and test-wasm-providers, neither of which depends on the test run. Move them to a cross-compile-checks job that runs alongside it. build-and-push-image waited on all four test jobs before starting even though it only pushes untagged per-arch digests. Let it run in parallel and put the lint/test gate on the merge-and-push-* jobs, which are the only ones that move tags. Together this cuts the main pipeline from ~13 min to ~8 min.
setup-go's built-in cache is keyed on go.sum alone, so one snapshot per OS was saved on 2026-08-31 and every job since restored that stale copy with "cache hit, not saving". Every job recompiled the whole dependency tree: build 2:09, golangci-lint 2:24, cross-compile 3:01, Windows tests 6:57. Replace it with a composite setup-go action that restores a per-job GOMODCACHE + GOCACHE keyed on <os>-<job>-<go.sum>-<sha> with prefix fallback, and save it at the end of each job on main only. PR runs restore from the latest main cache and never write, so cache churn stays bounded. Also stop running `go test ./e2e -run TestWasmProviderDependencies` in cross-compile-checks: `task test` already covers it via ./..., and cold it cost ~2 min just to compile the e2e package. Split the GOOS=js half into a test-wasm-js task for CI; test-wasm-providers keeps both for local use.
Build Cloud compiles amd64 and arm64 natively regardless of the GitHub runner's architecture, so the per-platform matrices only doubled runner count and setup overhead. Build both platforms in a single job per stage: build-image on PRs, build-and-push-image on main/tags. Pushing a multi-platform build by digest yields one untagged index, so the digest travels as a job output instead of per-arch artifacts, and the two merge jobs collapse into publish-tags, which still carries the lint/test gate and is the only job that moves tags. Disable setup-buildx's binary cache: the download takes seconds and the cache wrote a ~40 MB entry per PR branch, crowding the 10 GB repo limit. Net: 4 fewer runners per main run, 2 fewer per PR run, no artifacts.
aheritier
force-pushed
the
ci-improvements
branch
from
September 10, 2026 10:01
2641877 to
97339eb
Compare
aheritier
marked this pull request as ready for review
September 10, 2026 10:35
dgageot
approved these changes
Sep 10, 2026
aheritier
added a commit
that referenced
this pull request
Sep 10, 2026
Fork PRs receive neither repository variables nor an OIDC token, so the image job failed on `DOCKERHUB_OIDC_CONNECTION_ID must be configured` for the 60% of PRs that come from forks (the reason #4220 was reverted in #4223). The old condition compared the PR's *base* repository to this one, which is always true; trust is now decided on the *head* repository. Trusted runs keep Build Cloud and both platforms. Fork PRs build linux/amd64 on the runner with a plain docker-container builder, no login, nothing pushed, no build-record upload (the read-only token cannot create artifacts). The Dockerfile is exercised for every PR instead of being skipped.
aheritier
added a commit
that referenced
this pull request
Sep 10, 2026
…ally Fork PRs, PRs opened by Dependabot and other bots running with a read-only token, and CI in forks of this repo receive neither repository variables nor an OIDC token, so the image job failed on `DOCKERHUB_OIDC_CONNECTION_ID must be configured` (why #4220 was reverted in #4223). The old condition compared the PR's *base* repository to this one, which is always true. The job now decides at runtime: Build Cloud (both platforms, one runner) whenever the Hub OIDC connection id and the OIDC token endpoint are present, otherwise a plain docker-container builder on the runner, linux/amd64 only, nothing pushed, no build-record upload. Publishing runs still fail loudly when Build Cloud is unavailable rather than downgrading. The Dockerfile is exercised on every PR instead of being skipped or failing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A
mainrun took ~13–15 min and a PR run ~8–10 min. Per-step timings showed the time goes to compiling, not testing: every job recompiled the whole dependency tree because the Go build cache was effectively cold (see commit 2), and the image pipeline used 2× the runners it needed.What (one commit each)
Runners are left as-is: on a public repo
ubuntu-latest/windows-latestare already 4-core, and the org's larger runners are out of scope here.check-plan-cross+test-wasm-providersmove to across-compile-checksjob;build-and-push-imageno longer waits on tests (it pushes untagged digests only), the gate moves to the tag-publishing job.setup-go's built-in cache is keyed ongo.sumalone, so one snapshot per OS was saved on 2026-08-31 and every job since loggedCache hit … not saving cache. New composite.github/actions/setup-gorestoresGOMODCACHE+GOCACHEkeyed<os>-<job>-<go.sum>-<sha>with prefix fallback; saved onmainonly. Also drops the redundantgo test ./e2e -run TestWasmProviderDependenciesfrom CI (already covered bytask test; cold it cost ~2 min to compile e2e).publish-tags.cache-binary: falseon buildx stops a ~40 MB cache write per PR branch.Measured so far
PR run 34463832719 vs the last run on the previous layout (34458544304). Caches are not seeded yet (saves happen on
mainonly), so every job here starts cold — this is the floor, not the target.mainseeds a cacheSet up Go2:19 → 0:23 (no stale 944 MB restore);Run tests5:46 → 9:58 coldAfter merge: the first
mainrun seeds one cache per job; the next PR run is the real measurement (expected: build ~20 s, lint <1 min, cross-compile <1 min, Windows tests ~3–4 min). Windows cache size is the thing to check — if it lands well above 1 GB the restore eats into the gain.Tradeoffs
mainrun saves 5 caches (~0.5–1 GB each). The repo is at the 10 GB cap (503 caches, mostly per-PRbuildx-dl-bin-*), so stale entries should be purged once this lands.Follow-ups (repo settings, not in this PR)
cross-compile-checksandwindows-teststo themainruleset's required checks (currently onlybuild-and-test,license-check,lint).gh cache list --limit 500 --json id,key --jq '.[] | select(.key|test("^(buildx-dl-bin|setup-go-)")) | .id' | xargs -n1 gh cache delete🤖 Generated with Claude Code