[Doddanna17]

Context

npm processes publishes asynchronously (publish-time malware scanning, changelog 2026-07-28 — https://gh.io/npm-publish-time-scanning). A successful npm publish now only means the upload was accepted; the version becomes visible on the registry later — measured 2.9 min typical, 18.3 min and 55 min outliers, up to 24 h per npm's own email. Our release pipeline assumed immediate visibility.

Oct 17 (master 91eeaace5), five publish attempts, three failed: a hard-coded sibling wait expired at 19.8 min while @bitgo-beta/sdk-coin-tao@1.0.0-beta.1477 was still being processed (its successor surfaced 3 seconds after the poll gave up); reruns collided with already-published versions (TLOG_CREATE_ENTRY_ERROR (409), npm E403 cannot publish over the previously published versions); and runs reported green while packages were missing. Root cause confirmed with Infra in INF-4737 — the fix belongs in our publish scripts.

Change

scripts/verify-release.ts — the registry is now the source of truth, not the publish exit code:

• waits for the exact version to appear (GET /<pkg>/<version>, bounded by BITGO_PUBLISH_VISIBILITY_TIMEOUT_MINUTES, default 60) after publishing;
• reconciles a publish conflict when the version is already published and usable, instead of publishing the same content under a bumped version;
• classifies npm publish-over failures (E403 cannot publish over the previously published versions, EPUBLISHCONFLICT) as recoverable conflicts;
• requires bitgo to carry shrinkwrap metadata (_hasShrinkwrap) before counting it as published;
• warns when the @<preid> dist-tag has not moved to the published version;
• fails the run in every mode when a package cannot be published — previously a non-recovery run logged the failure and passed.

package.json — test:scripts runs the TypeScript suites via --import=tsx. It previously matched only *.test.js, so the shrinkwrap and prepare-release suites never executed in CI (the new suite would have been equally dead).

scripts/tests/verify-release.test.ts — new, 22 cases: conflict classification, visibility polling and budget, reconcile-vs-bump, bitgo shrinkwrap requirement, dist-tag warning, private-module skip, fail-closed.

Both release workflows — explicit visibility budgets (BITGO_SIBLING_AVAILABILITY_TIMEOUT_MINUTES=120, BITGO_PUBLISH_VISIBILITY_TIMEOUT_MINUTES=120) with timeout-minutes: 180 above them (the stable release job was hard-capped at 60, so a larger wait could not have been used), and BITGO_GENERATE_SHRINKWRAP=true on Verify Publish so a recovery re-publish of bitgo is not shipped without a shrinkwrap.

Why reconcile rather than bump

Bumping republishes identical content under a new version — that produced @bitgo-beta/sdk-coin-stx beta.2334 → .2335 → .2336 in 50 minutes, and it invalidates the shrinkwrap built for bitgo's previous version. When the registry already holds the intended version (npm E403), waiting for it to become visible is the correct resolution; a bump is only justified when the version is genuinely taken by different content (a TLOG conflict whose upload never landed).

Point-by-point rationale in this comment: how each change maps to the failure it fixes, why this approach was chosen over raising the timeouts / continuing to bump versions / removing the registry from the path, and why the pipeline sometimes passed anyway (the lag is a heavy-tailed race against a fixed 20 min wait inside a 60 min job cap).

Commits

SHA	Subject	Contents
8351328b63	fix(scripts): reconcile publish conflicts and wait for visibility	verify-release.ts, verify-release.test.ts, package.json, publish.yml, npmjs-release.yml

Tests

• New scripts/tests/verify-release.test.ts (22 cases).
• yarn test:scripts now also executes the previously-dead generate-bitgo-shrinkwrap.test.ts (14 cases) and prepare-commit-msg.test.js (12).

Verification

• Local: yarn test:scripts → 48 passing / 0 failing
• Local: npx tsc -p scripts/tsconfig.json → no errors in the changed files (only pre-existing key-card / sdk-core project errors)
• Local: npx prettier --check clean; npx eslint 0 errors; commitlint passed; commit GPG-signed
• CI on 8351328b63: code-quality (scripts), (format), (lint), (dependencies), (commit-lint) all pass — the previously-dead TypeScript suites now execute and are green
• Workflow diffs carry no unrelated formatting churn: the only deletion is timeout-minutes: 60; everything else is additions

Review follow-ups

None yet.

Related

• SI-2040 (this ticket), WCI-1858 (original analysis + PR fix(scripts): wait for workspace siblings to appear on npm before publishing bitgo #9951 partial fix), WCI-1754 (same lag hitting the stable "Verify recovery published the missing versions" step), INF-4737 (registry-side evidence: measured gaps, run IDs, npm async-processing confirmation)

Not in this PR (follow-ups)

• Build the bitgo shrinkwrap from local sibling tarballs so it never consults the registry (WCI-1858 fix Add README for examples directory #1).
• Decouple "is it live?" verification from the publish job into a separate retryable job/alert, so a slow registry cannot block the release.
• scripts/tests/prepareRelease/prepare-release-main.test.ts needs full git history (git clone --local + checkout of a historical commit); the CI job that runs test:scripts checks out shallow, so it stays excluded from the glob — it should move to a job with full history.
• The two hard-coded visibility loops in npmjs-release.yml ("Verify recovery published the missing versions" — single pass, no retry; "Verify bitgo package has shrinkwrap metadata" — 5 × 5 s) still need a real budget (WCI-1754).

[linear-code]

SI-2040

[Doddanna17]

How each change maps to a specific failure

Detail behind the diff, so a reviewer can check each claim against the Oct 17 symptoms rather than the summary.

The underlying assumption was "npm publish succeeded, therefore the version exists". Since npm's publish-time malware scanning (2026-07-28) that is false: exit 0 means the upload was accepted, and visibility follows later (2.9 min typical, 18.3 / 55 min observed, up to 24 h per npm's own email). The PR inverts that assumption in both directions — the registry becomes the source of truth, and the job fails loudly instead of reporting green.

1. Fail closed in every mode — scripts/verify-release.ts (verify())

Before, a failed package hit if (!RECOVERY_MODE) return;: a normal (non-recovery) release logged Failed to verify outstanding packages. and returned without setting an exit code. That is precisely how a run finished green with packages absent from npm. Now it walks every package, collects the failures, and always sets process.exitCode = 1. Effect: a missing package cannot be masked by a green job, and the log names every missing package instead of stopping at the first.

2. Reconcile instead of bump — publishWithRecovery()

Before, any conflict was answered by inc-ing the version and republishing identical content under a new number — the direct cause of @bitgo-beta/sdk-coin-stx beta.2334 → .2335 → .2336 in 50 minutes, and it invalidates the shrinkwrap already built for bitgo's previous version. Now a conflict first calls waitForVersionInfo(json.name, json.version); if the intended version is already published and usable, the conflict is resolved, not retried with a bump. Bumping is justified only when the version is genuinely taken by different content.

3. Recognise npm publish-over — isRecoverablePublishConflict()

Before, only transparency-log conflicts were recognised and the whole check sat behind !RECOVERY_MODE, so in normal mode any publish error threw immediately. Now EPUBLISHCONFLICT, cannot publish over the previously published versions and E403 … previously published are recoverable, and matching also inspects e.message (not just stdout/stderr), which is where npm's message landed in the failing runs. Effect: these errors route into the reconcile path above instead of aborting the release.

4. Verify visibility, not exit codes — waitForVersionInfo() + getVisibilityTimeoutMs()

Before, nothing confirmed the version ever appeared, and the sibling wait had a fixed budget a normal tail blew straight through (expired at 19.8 min while @bitgo-beta/sdk-coin-tao@1.0.0-beta.1477 was still being processed; its successor surfaced 3 seconds after the poll gave up). Now the code polls GET /<pkg>/<version> until visible, bounded by BITGO_PUBLISH_VISIBILITY_TIMEOUT_MINUTES (default 60, CI 120), and warns when the @<preid> dist-tag has not moved. Effect: decisions key off what consumers will actually resolve, and the measured outliers fit inside the budget.

5. bitgo must carry a shrinkwrap — isUsablePublishedVersion()

A published bitgo without _hasShrinkwrap no longer counts as published (logs "on npm but without shrinkwrap metadata", returns false). Effect: blocks the silently-broken case where bitgo ships without a shrinkwrap and consumers lose the pinned workspace siblings.

6. Budgets sized for npm's tail — both workflows

BITGO_SIBLING_AVAILABILITY_TIMEOUT_MINUTES=120 (consumed by scripts/generate-bitgo-shrinkwrap.ts while it waits for siblings), BITGO_PUBLISH_VISIBILITY_TIMEOUT_MINUTES=120, and job timeout-minutes: 60 → 180. Effect: the stable release job was hard-capped below the worst-case wait, so a slow-but-eventually-successful publish had its legs cut. The job now times out only if the waits themselves are exceeded, and the waits are the thing being reasoned about.

7. Recovery re-publish keeps the shrinkwrap — publish.yml Verify Publish

Adds BITGO_GENERATE_SHRINKWRAP: true. Without it a recovery re-publish of bitgo produced a tarball with no shrinkwrap — exactly the state change 5 now rejects. The two have to ship together.

8. The TypeScript suites were never running — package.json

Before, test:scripts was mocha './scripts/tests/**/*.test.js'; the .js glob never matched the TypeScript suites, so generate-bitgo-shrinkwrap.test.ts and prepare-release-main.test.ts did not execute in CI at all — the new suite would have been equally dead. Now --import=tsx plus the *.test.ts glob. Effect: this is the meta-fix; the regression tests for everything above now actually run in code-quality (scripts).

9. New suite — scripts/tests/verify-release.test.ts (22 cases)

Conflict classification, visibility polling and budget parsing, reconcile-vs-bump, the bitgo shrinkwrap requirement, dist-tag warning, private-module skip, fail-closed. Enabled by injecting VerifyDeps (fetchVersionInfo, publish, sleep, getDistTags, getLernaModules) so the tests touch no network and publish nothing. Effect: the behaviour is pinned against silent reintroduction of "green with missing packages".

Why this approach, and why it sometimes passed anyway

The choice of fix. Three alternatives were on the table, and the shape of this PR is what is left after rejecting them:

• Just raise the timeouts. Insufficient on its own. npm's own guidance puts the tail at up to 24 h, so no budget is provably safe, and raising the number changes nothing about the deeper defect: the pipeline still trusted npm publish exit 0 instead of checking the registry. A larger budget alone would still have produced green runs with packages missing, just less often.
• Keep bumping the version on conflict. Rejected — it republishes identical content under a new number (beta.2334 → .2335 → .2336), invalidates the shrinkwrap built for the previous bitgo version, and grows the registry with duplicates. The conflict almost always means "already published, not visible yet", which a wait resolves and a bump does not.
• Remove the registry from the path entirely — build bitgo's shrinkwrap from the local sibling tarballs so nothing has to be visible first. This is the real elimination of the class of bug, but it is a substantially larger change to release-critical code; it stays WCI-1858 fix Add README for examples directory #1 and is deliberately out of scope here.

What remains is the smallest change that makes the pipeline both tolerant of the measured lag and truthful about state: poll the registry for the version, reconcile instead of bumping, size the wait budgets above the observed tail, and fail closed.

Why it sometimes worked. This is the important part, and it is consistent with the diagnosis rather than an argument against it.

The visibility delay is a random variable, not a constant: 2.9 min typical, 18.3 min and 55 min observed. The old pipeline raced it with fixed budgets — a 20 min sibling wait (scripts/generate-bitgo-shrinkwrap.ts:25, the 19.8 min expiry in the failing run is that budget elapsing) inside a job hard-capped at 60 min (npmjs-release.yml:217). On a day when the draw fell near the median, every version appeared well inside 20 min and the release went green. On a tail day, the wait expired with the package still being processed — the sdk-coin-tao version's successor surfaced 3 seconds after the poll gave up, which is the boundary being hit. Same code, same commit, different outcome: success was the race being won, not the pipeline being correct. That intermittency is the signature of a timing bug, and it is why the fix is polling and budgeting rather than correcting flawed logic.

Two further sources of flip-flopping amplify the impression of non-determinism:

• Retries made things worse. After a failed run, a rerun republished; by then the earlier version had often become visible, so npm returned E403 … previously published or a TLOG 409 — which the old code classified as a hard failure, because the recoverable-conflict check was gated behind RECOVERY_MODE. So the retry was more likely to fail than the original run, and the failures looked random rather than sequential.
• A silent-pass path existed. In non-recovery mode verify() logged Failed to verify outstanding packages. and returned without setting an exit code, so a run could finish green while a package was genuinely missing. Some "it worked" outcomes were that path, not a successful publish.

The consequence for the fix: changes 1–3 and 5–9 are deterministic — fail-closed, reconcile, conflict classification, the shrinkwrap requirement, and tests that now execute in CI. Change 4 and 6 are tolerance, not elimination: with a budget above the measured tail, the measured distribution no longer fails, but a lag beyond 120 min still would. Eliminating that residue means removing the registry dependency (WCI-1858 fix #1) or moving the liveness check out of the publish job (WCI-1754) — both tracked, both out of scope here.

Scope

This does not change npm's asynchronous processing (impossible) and is not the root-cause elimination — it makes the pipeline tolerant of the tail and truthful about state. The eliminations stay tracked separately: building the bitgo shrinkwrap from local sibling tarballs so it never consults the registry (WCI-1858 fix #1), and decoupling "is it live?" into a separate retryable job (WCI-1754).

Note on the workflow diffs

publish.yml and npmjs-release.yml contain no unrelated formatting churn. An earlier revision of this branch had normalised 5 pre-existing lines (quote style and comment spacing, picked up from a local prettier --write); that was reverted, so the workflow diff is one deletion (timeout-minutes: 60) plus additions.

[zahin-mohammad]

Request changes: this improves fail-closed behavior and npm visibility polling, but the recovery path still has four release-blocking correctness problems.

1. Existing-version reconciliation does not verify artifact contents. VersionInfo keeps only found and hasShrinkwrap (scripts/verify-release.ts:23-45), and an existing version is accepted after a publish conflict at scripts/verify-release.ts:179-186. In the incident this PR is meant to fix, npm's @bitgo-beta/sdk-coin-eth@4.4.1-beta.2098 has shasum 6d031ba72e0b2d2b50b33b94cbc6fafc0a795fc2, while run 39991955081 produced 00090cba8bee983846790ec88059432bc20e3c91. The PR would treat that different artifact as reconciled. Please compare the locally packed tarball's integrity with npm's dist.integrity/dist.shasum; an occupied version with different contents must get a fresh version.

2. A stale consumer-facing dist-tag explicitly returns success. When the exact version exists but beta points elsewhere, verification warns and returns true (scripts/verify-release.ts:225-242). The new test codifies that false green (scripts/tests/verify-release.test.ts:189-194). This is the reported failure mode: consumers using @beta do not receive the intended release. Verification must repair and re-check the tag, or fail. For this particular 2098, retagging is unsafe because of the digest mismatch above.

3. The late recovery bump invalidates the prepared dependency graph and shrinkwrap. Recovery mutates only the failed package's package.json (scripts/verify-release.ts:196-204). Normal preparation propagates version changes to every internal dependent (scripts/prepare-release.ts:129-145). The workflow has already built and published siblings and bitgo before this bump (.github/workflows/publish.yml:103-139), so dependents and bitgo's npm shrinkwrap can remain pinned to the unusable old version. A fresh version must return to graph-wide dependency reconciliation, rebuild the affected closure, regenerate shrinkwrap, and publish dependencies before bitgo.

4. Fallback publication cannot publish the private bitgo workspace. The fallback calls plain npm publish (scripts/verify-release.ts:47-49), while modules/bitgo/package.json:1-6 has private: true. WCI-1650 (1895a4c010) already documented that npm can silently skip this workspace with exit code zero and only a stderr warning; that fix temporarily removed/restored private and surfaced stderr. These tests create bitgo without private: true (scripts/tests/verify-release.test.ts:43-48, :257-279), so the regression is not covered.

Please also add regression tests for digest mismatch, stale-tag repair/failure, dependency and shrinkwrap propagation after a recovery bump, and a real private: true bitgo fallback. Until these are addressed, the workflow can report success with the wrong artifact/tag or publish an internally inconsistent package set.