Skip to content

ci: migrate cosign to v3 and dual-publish image signatures - #3382

Merged
SamMorrowDrums merged 3 commits into
mainfrom
sammorrowdrums-cosign-v3-signing-migration
Oct 2, 2026
Merged

SamMorrowDrums merged 3 commits into
mainfrom
sammorrowdrums-cosign-v3-signing-migration

Conversation

@SamMorrowDrums

@SamMorrowDrums SamMorrowDrums commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Upgrade cosign from v2.6.5 to v3.1.3 and publish both legacy signatures and native Sigstore bundles for the identical immutable image digest. Unmodified verification commands work with the tested v2/v3 clients; the user-facing verification documentation initially added by this PR has been removed as requested.

Why

Follow-up to #3370, which deferred v3 because its bundle defaults could break consumers. No issue is automatically closed.

Both v2.6.5 and v3.1.3 fix GHSA-fx35-mq7g-6g98. v3.1.3 was the latest stable checked on 2026-10-02. The original implementation retained only legacy signatures; this revision implements the user's explicit request for dual publication without a consumer command migration or extra repository docs.

What changed

  • Keep the existing SHA-pinned installer and select cosign v3.1.3.
  • Sign once per repository/index digest in two sequential, required steps: legacy cosign sign --yes --new-bundle-format=false --use-signing-config=false --registry-referrers-mode=legacy, then native cosign sign --yes, with v3's default bundle format and TUF-provided signing-service configuration.
  • Both steps use exactly "${REGISTRY}/${IMAGE_NAME}@${DIGEST}", with DIGEST from the same build output. Tags are aliases to that image, not separate subjects; signing once avoids accumulating identical bundles for every alias.
  • Preserve GitHub OIDC/Fulcio, transparency logging, existing permissions, digest signing, and PR publication/signing exclusions. Either signing step failing fails the job; native signing is skipped if legacy signing fails.
  • Correct the pre-existing misleading private-image comment: --yes also consents to public transparency-log disclosure.
  • Remove docs/verify-artifacts.md and all README/SECURITY additions introduced by this PR. Pre-existing docs are unchanged. The aggregate PR diff is now only .github/workflows/docker-publish.yml.

Only the Docker workflow uses cosign. The release workflow and .goreleaser.yaml use GitHub build-provenance attestations for archives/checksums, not cosign blob signatures; those remain unchanged.

Consumer impact and supported limits

Tested unchanged command behavior, not universal client compatibility: cosign v2.5.3, v2.6.5, and v3.1.3 all verify the same dual-signed digest without any format/discovery flag. This was proved both locally and against the real user-authorized GHCR test image with GitHub OIDC/Fulcio/Rekor checks enabled. v2.5.3 is an older compatibility-test client, not a recommendation to use an unpatched release.

Client/path Observed behavior on the dual-signed digest
v2.5.3, default cosign verify Verifies the legacy .sig signature.
v2.6.5, default cosign verify Auto-discovers and verifies the native bundle.
v3.1.3, default cosign verify Auto-discovers and verifies the native bundle.
v3.1.3, --new-bundle-format=false Explicitly verifies the legacy .sig signature.

Important evidence: v2.6.5 auto-discovery overrides even --new-bundle-format=false when native bundles exist. We did not mislabel that result as legacy verification. Its default image command verified legacy before the native pass; its verify-blob also cryptographically verified the exact retained legacy payload/signature downloaded from .sig storage after dual publication. The older v2.5.3 default image command and v3's explicit legacy path verified .sig directly while native bundles coexisted.

Native verification output is not byte-for-byte the legacy JSON: the native critical.type is https://sigstore.dev/cosign/sign/v1, and critical.identity.docker-reference includes the digest. Consumers parsing those values or supplying custom legacy certificate/root flags need separate compatibility review. Retaining .sig does not force v2.6.5 to choose it. Unsupported/untested versions, offline/custom trust setups, policy engines, and registries other than tested GHCR/local registry:3 are not claimed universally compatible.

The legacy pass retains the legacy image payload, .sig storage and Rekor v1 defaults. --use-signing-config=false is necessary as well as the format opt-out: v3 rejects legacy image signing with its default signing config unless a local bundle is supplied. Native signing uses protobuf-defined Sigstore bundles serialized as JSON, an OCI referring artifact, and default TUF signing-service discovery. On both tested registries discovery used the OCI referrers fallback tag, distinct from .sig. This does not assume the registry serves the native referrers HTTP API.

The real native bundle's media type was application/vnd.dev.sigstore.bundle.v0.3+json; its transparency entry was dsse v0.0.1, with inclusion promise/proof and a rekor.sigstore.dev checkpoint. The legacy entry was hashedrekord. Native defaults must not be described as necessarily selecting Rekor v2: current TUF configuration selected the observed service. --new-bundle-format remains supported but deprecated in v3.1.3. cosign verify does not accept the signing-only --registry-referrers-mode flag.

Research: v3 announcement, v3.1.3 release, signing-config validation, v3 discovery, v2.6.5 discovery, and OCI publication.

Real GHCR / GitHub OIDC / Rekor test

Passed on the reviewed commit 813e3fa03465d1aee39f70771349c15ae68bb839: authorized isolated branch dispatch, both signing steps successful.

The user explicitly authorized isolated branch/test publication. Before dispatch, the pinned metadata action and ref rules were inspected: branch dispatch cannot generate semver/tag aliases; edge requires default branch main; schedule is inactive; explicit latest is disabled for a branch; automatic latest is not emitted for branch/sha tags. Actual run metadata confirmed only:

ghcr.io/github/github-mcp-server:sammorrowdrums-cosign-v3-signing-migration
ghcr.io/github/github-mcp-server:sha-813e3fa

Both resolve to the same multi-platform index digest:

sha256:4a5a05533a67518a32fed05da34f1b64bf8d3bcc8148053eced9725a85aba77d

Read-only registry inspection confirmed a retained legacy .sig manifest with certificate/Rekor material and a separate OCI referrers fallback index containing the native bundle. Both signatures' subjects match the digest above. No latest, edge, main/nightly, or release/version aliases were published by this dispatch. No Git tags were pushed; the GoReleaser/release workflows were not triggered; nothing was merged.

Exact verification command, unchanged for all three tested client versions — passed with no insecure bypass flags:

IMAGE='ghcr.io/github/github-mcp-server@sha256:4a5a05533a67518a32fed05da34f1b64bf8d3bcc8148053eced9725a85aba77d'
IDENTITY='https://github.com/github/github-mcp-server/.github/workflows/docker-publish.yml@refs/heads/sammorrowdrums-cosign-v3-signing-migration'
ISSUER='https://token.actions.githubusercontent.com'
cosign verify --certificate-identity "$IDENTITY" --certificate-oidc-issuer "$ISSUER" "$IMAGE"

v3.1.3 also passed that command with --new-bundle-format=false, proving the real legacy certificate/log path remains valid. Default v2.6.5/v3.1.3 rejected the wrong refs/heads/main certificate identity and an incorrect OIDC issuer. Verification checked the exact digest, workflow/ref identity, issuer, certificate trust, and transparency material. An empty, isolated DOCKER_CONFIG was used for anonymous reads; user registry credentials/configuration were not modified.

The branch test now covers real GHCR publication, v3 keyless issuance, and public Rekor material, not just a generated-key proxy. Remaining release gate: confirm any additional supported consumers/policies and the actual release/prerelease ref identity through the normal maintainer-controlled release process. A prerelease tag/ref was not authorized or exercised by this branch test; native HTTP referrers API mode, key/log rotation, parallel same-digest writers, and injected public-service outages were not tested.

Partial publication and retry semantics

Publication is not atomic. Image aliases are pushed before signatures. If legacy signing fails, CI fails and the native pass is skipped; an unsigned image or partial legacy upload may already exist. If native signing fails, CI fails, but a valid legacy signature and possibly partial native upload/log entries may remain. No failure is masked with continue-on-error, always(), or success-shaped fallback.

Retry the signing commands for the recorded immutable digest, then verify both formats and the expected identity. Local tests injected native failure after the legacy pass and proved retry completes dual publication without changing the existing .sig manifest. Two full same-digest reruns and a second signing key preserved all previous signatures. Native bundles can accumulate on rerun; this is safe append/retention behavior, not exactly-once/idempotent artifact counts. A full workflow rerun may rebuild a different digest because metadata changes, so it is not guaranteed to repair the old partially signed digest. Do not delete prior .sig/referrer artifacts as an automatic cleanup or silently accept a partial publication.

MCP impact

  • No tool or API changes — publisher workflow only; toolsnaps unchanged and passing.
  • Tool schema or behavior changed
  • New tool added

Prompts tested (tool changes only)

  • N/A: no MCP tool changes.

Security / limits

  • No security or limits impact
  • Auth / permissions considered — same workflow OIDC and packages permissions; both passes issue ephemeral Fulcio certificates; real exact identity/issuer verification passed.
  • Data exposure, filtering, or token/size limits considered — two public signing/log entries per digest may be generated; public-log disclosure is unchanged. Only local disposable-key tests disabled tlog upload/verification, never production or real consumer verification.

Tool renaming

  • I am renaming tools as part of this PR (e.g. a part of a consolidation effort)
    • I have added the new tool aliases in deprecated_tool_aliases.go
  • I am not renaming tools as part of this PR — names/schemas unchanged.

Note: if you're renaming tools, you must add the tool aliases. For more information on how to do so, please refer to the official docs.

Lint & tests

  • Linted locally with ./script/lint — GOTOOLCHAIN=go1.26.8 TMPDIR="$PWD/bin/lint-tmp" sh script/lint: passed, 0 issues, using the branch's pinned toolchain/linter.
  • Tested locally with ./script/test — GOTOOLCHAIN=go1.26.8 sh script/test: passed full race suite/toolsnaps, after lint.
Exact command/check Result
actionlint -oneline -ignore 'label "ubuntu-latest-xl" is unknown' .github/workflows/docker-publish.yml Passed; only the pre-existing custom runner diagnostic excluded.
cosign sign --yes --help and legacy command with all three compatibility flags plus --help on v3.1.3 Passed flag parsing; both actual formats also signed locally and in the real dispatch.
python3 test_dual.py (session-only harness; checksum-verified upstream v2.5.3/v2.6.5/v3.1.3 binaries, generated keys, disposable loopback registry:3) Passed coexistence, exact signature subjects/storage, default discovery and verification, retained signatures across two reruns/second signer, native failure/retry, wrong-key rejection, and tampering with each format.
python3 test_workflow.py (session-only harness reading the actual YAML and executing its commands with a failing cosign stub under bash -eo pipefail) Passed: successful flow invokes both commands on identical digest; legacy failure exits 42 with native skipped; native failure exits 43; both steps exclude PRs and cannot continue on error.
python3 verify_public.py sha256:4a5a05533a67518a32fed05da34f1b64bf8d3bcc8148053eced9725a85aba77d (read-only session harness) Passed actual test-tag/digest equality, both stored formats, unmodified real v2/v3 verification, v3 legacy verification, and wrong identity/issuer negatives.
git diff cd502333 -- README.md SECURITY.md and file absence check Passed: original docs unchanged; new verification file removed.
git diff --check Passed.

Local signing commands used generated keys and only locally added --key test.key --tlog-upload=false; the native test also used --use-signing-config=false to avoid public service use. Local verification used --key test.pub --insecure-ignore-tlog. These tests did not pretend to cover production service discovery; the subsequent authorized real dispatch used the exact unmodified publisher commands with GitHub OIDC/Rekor and covered that gap.

The local tamper test replaced every legacy signature's bytes, then every native DSSE signature's bytes under its own discovery index, asserting cryptographic failure rather than merely missing signatures. Valid native signatures still verified when legacy was corrupted; corrupt native bundles were rejected by default v2.6.5/v3.1.3 without silently falling back to a valid legacy signature. Generated-key wrong-key checks failed in both formats. Local registries were stopped; fixtures are outside the repository. No tests require consumers to change their commands.

Fresh hosted CI for 813e3fa0: 17 passing checks, 1 failing lint check. Docker PR build, all Go platform builds, CodeQL, docs-check and MCP diffs passed; both PR signing steps were correctly skipped. The real isolated dispatch additionally passed both signing steps.

The failing PR lint run uses newer linter/go-github changes now on main and reports pre-existing govet inline/type-parameter-inference errors in pkg/github/discussions.go:102–104. Current main lint also fails with that same class of errors in untouched code; main's go-github v92 merge lint failed too. The aggregate PR modifies none of those code/linter surfaces. That independent base regression was reported to the dependency coordinator and must be resolved before merge; this PR does not suppress it or mix in unrelated Go changes.

script/generate-docs is not applicable: no MCP tools/toolsets changed. Live PAT-based e2e and a release-tag build were not run.

Docs

  • Not needed — user explicitly requested removing this PR's extra signing/verification docs. Engineering evidence, version limits, and rollout/rollback are preserved here instead; existing unrelated docs are untouched.
  • Updated (README / docs / examples)

Rollout plan

  1. Review the successful isolated branch dispatch and real consumer verification above. Confirm supported older clients/policy consumers and accept v2.6.5's native auto-discovery/output differences; do not claim universal command/output compatibility.
  2. Resolve the independent current-main lint regression and require applicable checks to pass before merge. Release maintainers should verify actual prerelease/release identity and any policy-specific tooling through the normal authorized release process; no additional tag/publication is automatically triggered by this PR work.
  3. Merge only after approval. Main pushes and scheduled Docker runs publish too, so merge itself starts rollout; monitor both required signing steps and verify both artifacts at the first main/nightly and next stable release digests.
  4. Assign an owner/exit criteria for removing legacy publication. Deprecated v3 compatibility flags, Rekor v1 availability, native bundles/referrer discovery, and any future v4 migration require an explicit follow-up decision. Keep both formats until that decision is made.

Rollback plan

For a native-publication regression, remove/disable the native step and keep legacy signing with the tested flags; if necessary restore the installer to security-fixed v2.6.5 as well (its CLI accepts those flags). Do not claim that removing the step removes already-published native bundles: v2.6.5 can still discover those and cannot be forced to legacy with the false flag. Preserve existing signature/referrer artifacts; do not delete production .sig/referrer tags automatically.

For an unsigned/partial publication, have an authorized maintainer restore the known-good publisher and re-sign the recorded affected immutable digest in the required format(s), then verify both subjects and identities. A full rebuild may produce a new digest and is not guaranteed to repair the old one. Do not repoint production aliases or push/modify Git release tags as automated rollback. No merge has been performed.

Maintainer decisions: supported consumer/policy matrix and native output compatibility, release-ref gate, independent lint fix, signature-retention/duplicate-bundle policy, and owner/deadline for eventually retiring legacy publication.

Keep legacy image payloads, .sig tags, and Rekor v1 service defaults. Document exact cosign v2/v3 image verification and unchanged release archive attestations.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@SamMorrowDrums
SamMorrowDrums marked this pull request as ready for review October 2, 2026 09:51
@SamMorrowDrums
SamMorrowDrums requested a review from a team as a code owner October 2, 2026 09:51
Copilot AI balanced review requested due to automatic review settings October 2, 2026 09:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

The production keyless-signing migration requires the planned maintainer-controlled prerelease validation before approval.

Review effort: Balanced
Findings: None

What changed in this PR

Migrates container signing to cosign v3.1.3 while preserving legacy v2-compatible signatures and documents artifact verification.

Changes:

  • Pins cosign v3.1.3 with legacy bundle, Rekor v1, and .sig storage flags.
  • Adds exact verification guidance for images and release archives.
  • Links the new guide from primary security documentation.
File Description
.github/​workflows/​docker-publish.yml Upgrades and configures cosign signing.
docs/​verify-artifacts.md Adds image and archive verification guidance.
README.md Links and summarizes verification guidance.
SECURITY.md Directs users to artifact verification instructions.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@SamMorrowDrums SamMorrowDrums changed the title ci: migrate cosign to v3 while preserving legacy image signatures ci: migrate cosign to v3 and dual-publish image signatures Oct 2, 2026
SamMorrowDrums and others added 2 commits October 2, 2026 12:39
Sign the same immutable digest in separate fail-fast steps. Retain legacy signatures and publish native bundles with v3 service defaults. Remove the verification documentation introduced by this PR as requested; keep compatibility evidence and rollout plans in the PR description.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@SamMorrowDrums
SamMorrowDrums merged commit 36e5c48 into main Oct 2, 2026
19 checks passed
@SamMorrowDrums
SamMorrowDrums deleted the sammorrowdrums-cosign-v3-signing-migration branch October 2, 2026 11:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants