ci: migrate cosign to v3 and dual-publish image signatures - #3382
Merged
Merged
Conversation
Keep legacy image payloads, .sig tags, and Rekor v1 service defaults. Document exact cosign v2/v3 image verification and unchanged release archive attestations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
The production keyless-signing migration requires the planned maintainer-controlled prerelease validation before approval.
Review effort: Balanced
Findings: None
What changed in this PR
Migrates container signing to cosign v3.1.3 while preserving legacy v2-compatible signatures and documents artifact verification.
Changes:
- Pins cosign v3.1.3 with legacy bundle, Rekor v1, and
.sigstorage flags. - Adds exact verification guidance for images and release archives.
- Links the new guide from primary security documentation.
| File | Description |
|---|---|
.github/workflows/docker-publish.yml |
Upgrades and configures cosign signing. |
docs/verify-artifacts.md |
Adds image and archive verification guidance. |
README.md |
Links and summarizes verification guidance. |
SECURITY.md |
Directs users to artifact verification instructions. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Sign the same immutable digest in separate fail-fast steps. Retain legacy signatures and publish native bundles with v3 service defaults. Remove the verification documentation introduced by this PR as requested; keep compatibility evidence and rollout plans in the PR description. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Upgrade cosign from v2.6.5 to v3.1.3 and publish both legacy signatures and native Sigstore bundles for the identical immutable image digest. Unmodified verification commands work with the tested v2/v3 clients; the user-facing verification documentation initially added by this PR has been removed as requested.
Why
Follow-up to #3370, which deferred v3 because its bundle defaults could break consumers. No issue is automatically closed.
Both v2.6.5 and v3.1.3 fix GHSA-fx35-mq7g-6g98. v3.1.3 was the latest stable checked on 2026-10-02. The original implementation retained only legacy signatures; this revision implements the user's explicit request for dual publication without a consumer command migration or extra repository docs.
What changed
cosign sign --yes --new-bundle-format=false --use-signing-config=false --registry-referrers-mode=legacy, then nativecosign sign --yes, with v3's default bundle format and TUF-provided signing-service configuration."${REGISTRY}/${IMAGE_NAME}@${DIGEST}", withDIGESTfrom the same build output. Tags are aliases to that image, not separate subjects; signing once avoids accumulating identical bundles for every alias.--yesalso consents to public transparency-log disclosure.docs/verify-artifacts.mdand all README/SECURITY additions introduced by this PR. Pre-existing docs are unchanged. The aggregate PR diff is now only.github/workflows/docker-publish.yml.Only the Docker workflow uses cosign. The release workflow and
.goreleaser.yamluse GitHub build-provenance attestations for archives/checksums, not cosign blob signatures; those remain unchanged.Consumer impact and supported limits
Tested unchanged command behavior, not universal client compatibility: cosign v2.5.3, v2.6.5, and v3.1.3 all verify the same dual-signed digest without any format/discovery flag. This was proved both locally and against the real user-authorized GHCR test image with GitHub OIDC/Fulcio/Rekor checks enabled. v2.5.3 is an older compatibility-test client, not a recommendation to use an unpatched release.
cosign verify.sigsignature.cosign verifycosign verify--new-bundle-format=false.sigsignature.Important evidence: v2.6.5 auto-discovery overrides even
--new-bundle-format=falsewhen native bundles exist. We did not mislabel that result as legacy verification. Its default image command verified legacy before the native pass; itsverify-blobalso cryptographically verified the exact retained legacy payload/signature downloaded from.sigstorage after dual publication. The older v2.5.3 default image command and v3's explicit legacy path verified.sigdirectly while native bundles coexisted.Native verification output is not byte-for-byte the legacy JSON: the native
critical.typeishttps://sigstore.dev/cosign/sign/v1, andcritical.identity.docker-referenceincludes the digest. Consumers parsing those values or supplying custom legacy certificate/root flags need separate compatibility review. Retaining.sigdoes not force v2.6.5 to choose it. Unsupported/untested versions, offline/custom trust setups, policy engines, and registries other than tested GHCR/localregistry:3are not claimed universally compatible.The legacy pass retains the legacy image payload,
.sigstorage and Rekor v1 defaults.--use-signing-config=falseis necessary as well as the format opt-out: v3 rejects legacy image signing with its default signing config unless a local bundle is supplied. Native signing uses protobuf-defined Sigstore bundles serialized as JSON, an OCI referring artifact, and default TUF signing-service discovery. On both tested registries discovery used the OCI referrers fallback tag, distinct from.sig. This does not assume the registry serves the native referrers HTTP API.The real native bundle's media type was
application/vnd.dev.sigstore.bundle.v0.3+json; its transparency entry wasdssev0.0.1, with inclusion promise/proof and arekor.sigstore.devcheckpoint. The legacy entry washashedrekord. Native defaults must not be described as necessarily selecting Rekor v2: current TUF configuration selected the observed service.--new-bundle-formatremains supported but deprecated in v3.1.3.cosign verifydoes not accept the signing-only--registry-referrers-modeflag.Research: v3 announcement, v3.1.3 release, signing-config validation, v3 discovery, v2.6.5 discovery, and OCI publication.
Real GHCR / GitHub OIDC / Rekor test
Passed on the reviewed commit
813e3fa03465d1aee39f70771349c15ae68bb839: authorized isolated branch dispatch, both signing steps successful.The user explicitly authorized isolated branch/test publication. Before dispatch, the pinned metadata action and ref rules were inspected: branch dispatch cannot generate semver/tag aliases;
edgerequires default branchmain; schedule is inactive; explicitlatestis disabled for a branch; automaticlatestis not emitted for branch/sha tags. Actual run metadata confirmed only:Both resolve to the same multi-platform index digest:
Read-only registry inspection confirmed a retained legacy
.sigmanifest with certificate/Rekor material and a separate OCI referrers fallback index containing the native bundle. Both signatures' subjects match the digest above. Nolatest,edge, main/nightly, or release/version aliases were published by this dispatch. No Git tags were pushed; the GoReleaser/release workflows were not triggered; nothing was merged.Exact verification command, unchanged for all three tested client versions — passed with no insecure bypass flags:
v3.1.3 also passed that command with
--new-bundle-format=false, proving the real legacy certificate/log path remains valid. Default v2.6.5/v3.1.3 rejected the wrongrefs/heads/maincertificate identity and an incorrect OIDC issuer. Verification checked the exact digest, workflow/ref identity, issuer, certificate trust, and transparency material. An empty, isolatedDOCKER_CONFIGwas used for anonymous reads; user registry credentials/configuration were not modified.The branch test now covers real GHCR publication, v3 keyless issuance, and public Rekor material, not just a generated-key proxy. Remaining release gate: confirm any additional supported consumers/policies and the actual release/prerelease ref identity through the normal maintainer-controlled release process. A prerelease tag/ref was not authorized or exercised by this branch test; native HTTP referrers API mode, key/log rotation, parallel same-digest writers, and injected public-service outages were not tested.
Partial publication and retry semantics
Publication is not atomic. Image aliases are pushed before signatures. If legacy signing fails, CI fails and the native pass is skipped; an unsigned image or partial legacy upload may already exist. If native signing fails, CI fails, but a valid legacy signature and possibly partial native upload/log entries may remain. No failure is masked with
continue-on-error,always(), or success-shaped fallback.Retry the signing commands for the recorded immutable digest, then verify both formats and the expected identity. Local tests injected native failure after the legacy pass and proved retry completes dual publication without changing the existing
.sigmanifest. Two full same-digest reruns and a second signing key preserved all previous signatures. Native bundles can accumulate on rerun; this is safe append/retention behavior, not exactly-once/idempotent artifact counts. A full workflow rerun may rebuild a different digest because metadata changes, so it is not guaranteed to repair the old partially signed digest. Do not delete prior.sig/referrer artifacts as an automatic cleanup or silently accept a partial publication.MCP impact
Prompts tested (tool changes only)
Security / limits
Tool renaming
deprecated_tool_aliases.goNote: if you're renaming tools, you must add the tool aliases. For more information on how to do so, please refer to the official docs.
Lint & tests
./script/lint—GOTOOLCHAIN=go1.26.8 TMPDIR="$PWD/bin/lint-tmp" sh script/lint: passed, 0 issues, using the branch's pinned toolchain/linter../script/test—GOTOOLCHAIN=go1.26.8 sh script/test: passed full race suite/toolsnaps, after lint.actionlint -oneline -ignore 'label "ubuntu-latest-xl" is unknown' .github/workflows/docker-publish.ymlcosign sign --yes --helpand legacy command with all three compatibility flags plus--helpon v3.1.3python3 test_dual.py(session-only harness; checksum-verified upstream v2.5.3/v2.6.5/v3.1.3 binaries, generated keys, disposable loopbackregistry:3)python3 test_workflow.py(session-only harness reading the actual YAML and executing its commands with a failing cosign stub underbash -eo pipefail)python3 verify_public.py sha256:4a5a05533a67518a32fed05da34f1b64bf8d3bcc8148053eced9725a85aba77d(read-only session harness)git diff cd502333 -- README.md SECURITY.mdand file absence checkgit diff --checkLocal signing commands used generated keys and only locally added
--key test.key --tlog-upload=false; the native test also used--use-signing-config=falseto avoid public service use. Local verification used--key test.pub --insecure-ignore-tlog. These tests did not pretend to cover production service discovery; the subsequent authorized real dispatch used the exact unmodified publisher commands with GitHub OIDC/Rekor and covered that gap.The local tamper test replaced every legacy signature's bytes, then every native DSSE signature's bytes under its own discovery index, asserting cryptographic failure rather than merely missing signatures. Valid native signatures still verified when legacy was corrupted; corrupt native bundles were rejected by default v2.6.5/v3.1.3 without silently falling back to a valid legacy signature. Generated-key wrong-key checks failed in both formats. Local registries were stopped; fixtures are outside the repository. No tests require consumers to change their commands.
Fresh hosted CI for
813e3fa0: 17 passing checks, 1 failing lint check. Docker PR build, all Go platform builds, CodeQL, docs-check and MCP diffs passed; both PR signing steps were correctly skipped. The real isolated dispatch additionally passed both signing steps.The failing PR lint run uses newer linter/go-github changes now on
mainand reports pre-existinggovetinline/type-parameter-inference errors inpkg/github/discussions.go:102–104. Current main lint also fails with that same class of errors in untouched code; main's go-github v92 merge lint failed too. The aggregate PR modifies none of those code/linter surfaces. That independent base regression was reported to the dependency coordinator and must be resolved before merge; this PR does not suppress it or mix in unrelated Go changes.script/generate-docsis not applicable: no MCP tools/toolsets changed. Live PAT-based e2e and a release-tag build were not run.Docs
Rollout plan
Rollback plan
For a native-publication regression, remove/disable the native step and keep legacy signing with the tested flags; if necessary restore the installer to security-fixed v2.6.5 as well (its CLI accepts those flags). Do not claim that removing the step removes already-published native bundles: v2.6.5 can still discover those and cannot be forced to legacy with the false flag. Preserve existing signature/referrer artifacts; do not delete production
.sig/referrer tags automatically.For an unsigned/partial publication, have an authorized maintainer restore the known-good publisher and re-sign the recorded affected immutable digest in the required format(s), then verify both subjects and identities. A full rebuild may produce a new digest and is not guaranteed to repair the old one. Do not repoint production aliases or push/modify Git release tags as automated rollback. No merge has been performed.
Maintainer decisions: supported consumer/policy matrix and native output compatibility, release-ref gate, independent lint fix, signature-retention/duplicate-bundle policy, and owner/deadline for eventually retiring legacy publication.