Skip to content

fix(website): upgrade Next to 16.4.6 to stop the dev server heap leak - #1610

Merged
blove merged 2 commits into
mainfrom
blove/website-next-dev-heap-leak
Nov 16, 2026
Merged

blove merged 2 commits into
mainfrom
blove/website-next-dev-heap-leak

Conversation

@blove

@blove blove commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Why

Website — e2e (control-plane) keeps losing its runner. It gets "The runner has received a shutdown signal" after 10–12 minutes, or "The hosted runner lost communication with the server" after about 55 minutes, and no test failure is reported. Recent cases: main at 1976eef49 (job 132945220886), #1598, #1599, #1607, #1608. #1602 lists six more from Nov 12–15.

Cause: the runner runs out of memory. Two things fill its 16 GB:

  1. About 10 GB of fixed cost. In fix(website): serve built Angular examples in the control-plane CI pool #1602's diagnostics run (job 132895652320), MemAvailable fell from 14.4 GB to 1.0 GB while the leg's nine Angular ng serve dev servers started (~1.1–1.3 GB each). It then stayed at 1–2 GB, with up to 3 GB in swap, for the whole suite.
  2. A leak in Next 16.2.12's dev server, which grows with each request. next-server reached 5.3 GB RSS by the end of that run. That is why the runner dies at 85–90% of the suite (193/228 on feat(ag-ui): project sources listed in state as message citations #1598, 333/371 on main). In the main job's log, the gap between tests stretches from 1 s to 37 s just before the shutdown, which is swap thrash.

The Website suite code has not changed since 213242e4a. This was memory at the margin, not a single commit.

Where the leak is

  • After a forced GC, a local next dev's heap kept rising when the same pages were requested again: about 1.2 MB per docs page render and 0.4 MB per /privacy render. /llms.txt (a route handler) retained nothing.
  • A heap snapshot diff over 150 requests showed +563k WeakRef, +610k Object and +1.2M heap numbers. The retainer is React Flight's dev-only async_hooks tracker: the pendingOperations Map → {promise: WeakRef, awaited, previous} chains. The live Map entries are Promises owned by ReadableStream tee/pipeTo branches (Node's forwardChunk pump) that never finish. Each one pins its request's whole await chain.
  • A two-file Next app with no repository code leaks the same way on 16.2.12 (heap 88 → 229 MB over 300 requests). On 16.4.6 it stays flat (87 → 92 MB).

What

  • Bump next from ~16.2.12 to ~16.4.6 in the root and Website package.json. The lockfile was updated with npm 10 (Node 22) against the existing lock, not regenerated. All 8 @next/swc-* platform entries are still there. The only other changes are Next's own dependencies (@next/*, sharp, postcss).
  • Pin back two defaults that 16.4 changed, so behaviour stays the same:
    • agentRules: false: otherwise next dev writes a generated apps/website/AGENTS.md. The repository already has its own contributor guide.
    • experimental.useTypeScriptCli: false: 16.4 runs the tsc CLI during next build, which also checks *.spec.tsx and applies project-reference file-list rules (185 errors, none new). This keeps the build's existing in-process type check. It neither widens nor narrows coverage. Moving to the CLI checker can be its own change.

No test, timeout, retry or workflow setting is changed.

Verification

Full local control-plane suite, same machine, with next-server heap logged after a forced GC every 15 s:

live heap at end failures duration
Next 16.2.12 (main) climbs 0.2 → 1.22 GB, still rising 4 failed 8.7 min
Next 16.4.6 (this PR) plateaus 0.38–0.42 GB 370 passed 6.8 min

On 16.4.6 the per-request probe on the real site is flat too: the docs page went 199 → 208 MB over 200 requests, against 624 → 781 MB on 16.2.12.

  • GROWTH_FORM_POLICY=growth_v1 npx nx build website: passes (2,185 static pages).
  • npx nx test website: passes. npx nx lint website: 0 errors.
  • next dev with this config does not write AGENTS.md.

Follow-up worth considering

The ~10 GB fixed cost of nine Angular dev servers stays. After this fix the leg should have several GB of headroom, but serving those apps from one build plus a single static/proxy server would remove most of it. That is a separate change.

🤖 Generated with Claude Code

The Website control-plane e2e leg keeps losing its runner. Nine Angular
dev servers take about 10 GB of the runner's 16 GB, and Next 16.2.12's
dev server then leaks heap on every rendered RSC page until the runner
runs out of memory near the end of the suite.

The leak is React Flight's dev-only async_hooks tracker: never-finished
stream pipe promises keep pendingOperations entries alive, and each one
pins its request's whole await chain. A two-file Next app on 16.2.12
leaks the same way; on 16.4.6 the heap stays flat.

Over a full local control-plane run, the next-server heap after GC rose
from 0.2 GB to 1.22 GB on 16.2.12 and plateaued at 0.38-0.42 GB on
16.4.6. The run also went from 4 failures in 8.7 min to 370 passed in
6.8 min.

Next 16.4 also changes two defaults, which this commit pins back:
- agentRules: next dev now writes a generated AGENTS.md into the app.
- experimental.useTypeScriptCli: next build now runs the tsc CLI, which
  checks spec files and project-reference file lists. Keep the existing
  in-process build type check.
@vercel

vercel Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
threadplane Ready Ready Preview Nov 16, 2026 12:57am UTC

Request Review

….4.6

The installed A2UI proof pins a SHA-256 of package-lock.json so that any
dependency change is reviewed against it. The Next 16.4.6 upgrade
changes the lockfile, so pin the new hash.
@blove
blove added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 589583e Nov 16, 2026
132 checks passed
@blove
blove deleted the blove/website-next-dev-heap-leak branch November 16, 2026 09:59

This branch was successfully deployed

1 active deployment
Preview – threadplane — 64a77ac1 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant