Skip to content

fix(build): publish the attested tarballs to npm - #2939

Closed
rytimx wants to merge 3 commits into
mainfrom
fix/publish-attested-tarballs
Closed

fix(build): publish the attested tarballs to npm#2939
rytimx wants to merge 3 commits into
mainfrom
fix/publish-attested-tarballs

Conversation

@rytimx

@rytimx rytimx commented Aug 3, 2026

Copy link
Copy Markdown

Problem

Since the nx 19 → 23 upgrade (#2928), the tarballs published to npm no longer match the tarballs signed by the Generate SDK Package Attestations step, so attestation verification (gh attestation verify) fails for the published artifacts of every release since 2.24.0.

Root cause: the workflow attests the output of pnpm pack-npm-packages, but nx release publish re-packs each package at publish time. nx 23 serializes the package.json dependencies map in project-graph order rather than the repo file's (alphabetical) order, so the two tarballs differ byte-wise for any package with enough internal workspace:* dependencies for the ordering to change. Affected on every release since 2.24.0: @imtbl/sdk, @imtbl/passport, @imtbl/minting-backend, @imtbl/checkout-sdk, @imtbl/checkout-widgets (≥5 internal deps each). Packages with ≤2 internal deps coincidentally still match.

Verified evidence:

  • Attestation 37650781 records imtbl-sdk-2.24.0.tgz = 39b7fc1e…, while the registry serves 01243a75…; @imtbl/config@2.24.0 matches its attested digest exactly.
  • The publish step's own npm notice shasum matches the registry bytes — the divergence is pack-vs-publish within a single run.
  • Alpha releases (which skip the changelog step) mismatch identically, ruling out everything except the re-pack inside nx release publish.
  • Registry package.json for @imtbl/sdk@2.24.0 lists dependencies in project-graph order; the repo file and pnpm pack output are alphabetical.

Change

  • Release to NPM now publishes the exact tarball files that were packed and attested (npm publish <tgz>), in topological order, instead of letting nx re-pack. Byte identity between attested and published artifacts is guaranteed by construction, so no future toolchain upgrade can silently reintroduce this drift.
    • Skips versions that already exist on the registry (safe re-runs after partial failures).
    • Keeps --tag alpha|latest and dry_run behaviour, and fails loudly if any packed+attested tarball is left unprocessed.
    • npm OIDC trusted publishing and NPM_CONFIG_PROVENANCE behave the same for tarball publishes.
  • New Verify Published Artifacts Match Attested Tarballs step: after publishing, downloads each published tarball from the registry and asserts its SHA256 equals the attested file (with retries for registry propagation), closing the loop end-to-end.

Testing

  • YAML and shell syntax validated locally; the topological-sort script was exercised against real registry tarballs (@imtbl/config correctly ordered before its dependent @imtbl/sdk).
  • Validation gate before the next public release (note: --dry-run does not exercise sigstore signing, so provenance must be proven on a real publish):
    1. One workflow_dispatch with dry_run: true — exercises listing, ordering, and the ambiguity check. Without a version bump every package reports Skipping … - already published; that is the expected green outcome.
  • Already tested locally against the real 22-tarball 2.24.4 set: topological ordering correct across all packages, dry-run exits green (all skipped), the publish path exercises npm publish --dry-run for unpublished versions, and the verify step passes on matching bytes / fails loudly on a tampered tarball. The integrity checker's own Go verification code was also run live against the registry: all 12 packages that already carry pnpm-pack bytes verify successfully, confirming checker compatibility with this pipeline's output.
    2. Let the next merge to main publish an alpha — this runs the identical publish + verify code for real.
    3. Confirm provenance exists for the new alpha via https://registry.npmjs.org/-/npm/v1/attestations/@imtbl%2Fsdk@<version> (2 attestations expected).
    4. Only then cut a public release.
  • Runner npm version note: if the runner picks up npm 12.x, ensure ≥ 12.0.1 (12.0.0 had a sigstore packaging bug, [BUG] npm publish results in Cannot find module 'sigstore' npm/cli#9722).

Impact

External consumers: none. Versions, dist-tags, npm install behaviour, existing lockfiles, npm provenance, and the jsDelivr CDN are all unchanged. The only observable difference is cosmetic: the published package.json dependency list returns to alphabetical order (matching the repo file, as it was pre-2.24.0). JSON key order has no runtime meaning and nothing (npm, bundlers, lockfiles) depends on it.

Release process: same triggers and inputs. release_type/dry_run behave as before; the npm_release step id is preserved so Slack notifications are untouched; the loop replicates nx's topological publish order and skip-if-already-published (which also makes re-runs after a partial failure safe). Alphas on every merge to main exercise the new path daily.

Known risks, both bounded and fail-safe (worst case is a failed release run, never a wrong artifact on npm):

  • npm publish <tarball> --provenance should be confirmed once via dry_run and an alpha, since tarball publishes have had npm quirks historically.
  • The verify step retries up to ~2.5 min/package for registry propagation — same tolerance the existing "Wait for NPM" step already assumes.

Notes

  • Versions 2.24.0–2.25.x are immutable on npm and will keep failing gh attestation verify; this fix is forward-looking from the next release. npm's own provenance still covers those versions.
  • Maintenance offer: if this team would rather not own the publish loop long-term, prodsec is happy to extract pack → attest → publish → verify into a versioned composite action (or reusable workflow) that we maintain, and this file shrinks back to a few lines. Raise it in review and we'll take that as a follow-up.
  • Tool context: the attested tarballs come from the Pack step (pnpm pack-npm-packages); the divergence was introduced by the re-pack inside nx release publish after the nx 19→23 upgrade — this PR removes that second packing path entirely.

🤖 Generated with Claude Code

Since the nx 19 -> 23 upgrade (#2928), nx release publish re-packs each
package at publish time and serializes package.json dependencies in
project-graph order, while the attestation step signs the pnpm pack
tarballs (which preserve the repo file's key order). Packages with
enough internal workspace deps get reordered, so the published tarball
SHA256 no longer matches any attested subject and attestation
verification fails for the published artifacts (every release since
2.24.0: @imtbl/sdk, passport, minting-backend, checkout-sdk,
checkout-widgets).

Publish the exact tarballs generated by pack-npm-packages and signed by
actions/attest instead of letting nx re-pack, and verify post-publish
that the registry serves the attested bytes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@nx-cloud

nx-cloud Bot commented Aug 3, 2026

Copy link
Copy Markdown

View your CI Pipeline Execution ↗ for commit 8539cc4

Command Status Duration Result
nx affected -t build,test ✅ Succeeded <1s View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗


☁️ Nx Cloud last updated this comment at 2026-08-03 01:32:20 UTC

rytimx and others added 2 commits August 3, 2026 09:08
Address adversarial review findings:
- Derive the publish list from the attested tarballs themselves
  (tar -xzOf package/package.json) with an explicit topological sort,
  instead of parsing pnpm -r exec output which interleaves progress
  and banner noise on stdout.
- Guard npm view/publish against consuming loop stdin.
- Replace the count assertion with a clearer ambiguity check
  (duplicate or unreadable tarballs refuse to publish, with a
  message naming the invariant).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by local end-to-end testing against the real 2.24.4 tarball set:

- npm publish --dry-run still validates against the registry and errors
  on already-published versions, so the already-published skip check now
  runs in dry-run mode too (a dry_run dispatch without a version bump
  reports every package as Skipping and exits green).
- npm view stdout is not guaranteed pure (update notices, wrapper
  banners), so the verify step extracts the tarball URL defensively.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@rytimx
rytimx marked this pull request as ready for review August 3, 2026 01:34
@rytimx
rytimx requested review from a team as code owners August 3, 2026 01:34
@rytimx rytimx closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants