[rush] shrinkwrap-deps.json grew ~100x per project since 5.179.0 (#5702); proposal: one aggregate digest per workspace dependency #6164

Summary

I was upgrading a large pnpm-workspaces monorepo from Rush 5.178.x to 5.181.0. After rush install, every project's .rush/temp/shrinkwrap-deps.json had grown by two orders of magnitude: one representative project's file went from 433 bytes to 154 KB, and the 1,498 files together total 237.8 MiB, up from 25.2 MiB.

The cause is #5702 (Rush 5.179.0). It fixed a real gap: PnpmShrinkwrapFile.getIntegrityForImporter used to drop workspace (link:) dependencies, so a change deep in a linked project did not change the dependent project's file. It fixed this by recursively merging every linked project's integrity entries into the dependent's file, so each file now carries the external dependency tree of its whole workspace closure, repeated once per dependent.

This may be the same root cause as #6129, which reports that an up-to-date rush install on ~1,800 projects spends ~10.8 s of CPU in updateProjectShrinkwrapAsync with 5.181.0. Building the merged maps is exactly the work that grows with the closure size, so the regeneration cost in #6129 and the file sizes here are likely two symptoms of #5702's merge strategy.

Repro steps

1. Take a Rush monorepo with useWorkspaces: true and projects that depend on each other with workspace:* (the repo below has 1,498 projects and a 7 MB pnpm-lock.yaml).
2. With rushVersion 5.178.x, run rush install and note the size of <project>/.rush/temp/shrinkwrap-deps.json for a project with workspace dependencies.
3. Set rushVersion to 5.181.0 (any version from 5.179.0 on), run rush install, and compare.

Expected result: the file gains sensitivity to workspace dependency changes at a cost roughly proportional to the number of workspace dependencies.

Actual result: the file contains the merged entries of every project in its workspace closure. Measured on the repo above, warm state:

Sizes and times come from regenerating every project's file with the given rush-lib version over the same pnpm-lock.yaml, the way WorkspaceInstallManager.postInstallAsync does (concurrency 10); times are the median of 3 runs.

The files are regenerated on every rush install, including an up-to-date one, and are read by ProjectChangeAnalyzer and the build cache, so the size shows up in install time, in change-detection hashing, and in disk usage under .rush/temp.

Details

Cause. getIntegrityForImporter resolves each link: dependency to its importer key, calls itself recursively, and merges the whole returned map into the caller's map. For a project with a deep workspace closure, that is the union of every external package in the closure, and it is repeated in every dependent.

Proposed fix. Keep the transitive sensitivity introduced by #5702, but stop copying entries. For each direct workspace (link:) dependency, record exactly one entry in the existing ${key}:${sha256Base64}: shape, where the digest summarizes the linked project's transitive dependency state:

• Compute a local map per importer: its own entry, its external packages' entries, and the set of importer keys reached through link: specifiers (both in the importer section and inside package snapshots).
• The aggregate digest for a linked project collects its link: closure into a set and hashes the local digests of the members in sorted key order. This makes it independent of traversal order and safe for A -> B -> A cycles.

Measured on the same repo, same method, with a branch implementing this:

The remaining growth over pre-5.179 is one entry per direct workspace dependency.

Compatibility. Any change in a linked project's closure still changes the dependent's aggregate digest, so hasChanges, git: selectors and rush change keep #5702's semantics. Upgrading rewrites every project's file once, so build-cache keys roll once, exactly as they did when 5.179.0 shipped. No new configuration or experiment flag; non-link entries are unchanged.

Alternatives considered.

• Storing the literal union of the closure's entries instead of a digest of local digests. Same sensitivity, since one key always has one value within a lockfile, but it keeps the files large.
• Skipping regeneration or saving with onlyIfChanged, as proposed in [rush] Up-to-date install still regenerates and rewrites every project's shrinkwrap-deps.json (~17 s per rush install/start) #6129. Those reduce how often the files are written; this proposal reduces what they contain and how long each generation takes (1.8 s to 0.4 s here, compared with 0.3 s before fix: Remove external filter in PnpmShrinkwrapFile.getIntegrityForImporter #5702). If [rush] Up-to-date install still regenerates and rewrites every project's shrinkwrap-deps.json (~17 s per rush install/start) #6129's ~17 s has the same cause, this change should remove most of it; the two fixes are independent and could still both land.

I have a branch with the implementation and updated unit tests (including a cycle test in both traversal orders) and will open a PR if this design is acceptable.

Standard questions