test: fix six recurring CI flakes by codebytere · Pull Request #65780 · nodejs/node

codebytere · 2026-09-04T08:27:08Z

Six CI flakes on main, each root-caused and fixed in its own commit. Four keep showing up in the nodejs/reliability reports over the last two weeks (picked by counting distinct main-targeting PRs they failed in since 2026-08-21, dropping everything since fixed on main or already in review in #65755, #65767, #65770); the last two (test-dns-resolver-max-timeout, the node:bench tests) failed this PR's own CI runs this week and will be in the next reports.

test	where	why it fails
`client-proxy/test-https-proxy-request-invalid-char-in-url`	macOS	only client-proxy test asserting zero proxy socket errors; the client resets the tunnel while the proxy is still relaying the upstream's TLS shutdown, logged as `ECONNRESET`
`parallel/test-external-memory-reasonable-size`	SmartOS	the child aborts with 1.2 GB resident and writes a core file, which outlasts the 300 s timeout; now runs under `ulimit -c 0` like the other abort tests
`test-runner/test-run-watch-emit-restarted`	macOS x64 (marked flaky)	FSEvents reports the fixture setup writes after the first run has started, so the intentional write is the second restart; now only restarts after the write count, flaky mark dropped
`wasi/test-wasi-pthread`	all (marked flaky on three platforms)	the fixture's `thread-spawn` waits on the same value the worker stores on success, so an early notify is lost, and gives the worker only 1 s to instantiate; either way `pthread_create()` fails. Flaky marks dropped
`parallel/test-dns-resolver-max-timeout`	rhel10-ppc64le	compares the wall time of `{ timeout: 500, tries: 3 }` with and without `maxTimeout: 500`, but c-ares only expires a try on cares_wrap's `timeout` ms timer tick (one or two ticks per try) and jitters uncapped retries by 0.5-1x, so the two ranges meet at 3000 ms; now `timeout: 100, tries: 5`, where they are ~600-1000 ms vs ~5000-7500 ms
`parallel/test-bench-*`	debian12-x64	the tests record samples with `b.start(); process.hrtime.bigint(); b.end(1)` and rely on the middle read to move the clock; on that host all three reads return the same value and `end()` throws `insufficient clock precision`. Now a `completeSample()` helper in `test/common/bench.js` spins until hrtime moves, used at every site and in the bench-runner fixtures

Looked at and left alone: parallel/test-runner-run ("should support timeout" reporting uncaughtException on macos15-x64) doesn't reproduce under load on Linux and I couldn't pin it down from the code; sequential/test-debugger-pid and test-run-watch-cwd-isolation-none* need a Windows/macOS box; pummel/test-fs-watch-non-recursive on AIX is the host running out of AHAFS watchers.

Tests: each changed test passes locally (Linux x64), 16-48 repeats at -j16 for the watch and WASI ones, 40 concurrent runs beside 24 busy-loop processes for the dns one, all test-bench-* under a preload that quantizes process.hrtime.bigint() to 100 µs (six of them fail that way before the change), and the parallel/ ones also under tools/test.py --worker.

Fixes: #64226
Refs: #59146
Refs: #54534

Disclosure: the code and this description were written by Claude Code, directed and reviewed by @codebytere.

nodejs-github-bot · 2026-09-04T08:27:14Z

Review requested:

@nodejs/wasi

codecov · 2026-09-04T10:12:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.20%. Comparing base (b0d56c7) to head (b26f47e).
⚠️ Report is 14 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #65780   +/-   ##
=======================================
  Coverage   90.19%   90.20%           
=======================================
  Files         770      770           
  Lines      264410   264410           
  Branches    50243    50236    -7     
=======================================
+ Hits       238490   238516   +26     
+ Misses      16924    16907   -17     
+ Partials     8996     8987    -9

see 31 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

panva · 2026-09-04T15:20:18Z

cc @nodejs/build this jenkins run started by @codebytere does not have Resume CI

MikeMcC399 · 2026-09-04T16:23:26Z

It's similar to the problem I also reported in nodejs/build#4453

17:15:25 FATAL: Channel "hudson.remoting.Channel@39c0b532:JNLP4-connect connection from 67.158.54.159/67.158.54.159:54594": Remote call on JNLP4-connect connection from 67.158.54.159/67.158.54.159:54594 failed. The channel is closing down or has closed down

gurgunday

lgtm

lpinca · 2026-09-05T17:40:08Z

There is another PR (#65770) for test/parallel/test-child-process-fork-closed-channel-segfault.js` that was opened before this one.

test-https-proxy-request-invalid-char-in-url is the only client-proxy test that asserts the proxy logged no socket errors at all. Once the last response has been read the client destroys its tunnel, and if the proxy is still relaying the upstream's TLS close_notify at that point the client answers with a reset, which the proxy records as ECONNRESET on the CONNECT socket. That has been failing the test on macOS even though every request was routed to the sanitized URL. Keep asserting on other errors but leave connection resets out. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

test-external-memory-reasonable-size makes a child allocate 1.2 GB of external memory so that V8's --external-memory-max-reasonable-size check fires and the process aborts. The abort raises SIGABRT with all of that memory resident, and on hosts that write core files (the SmartOS CI machines in particular) the dump takes longer than the test timeout, so the test has been timing out there since it was added. Run the child under `ulimit -c 0` on POSIX, the same way test-abort-fatal-error and common.childShouldThrowAndAbort() handle their aborting children. Refs: nodejs#65589 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

test-run-watch-emit-restarted expected exactly one test:watch:restarted event, but it starts run({ watch: true }) right after writing the fixtures into the watched directory. Watch backends that deliver events with some latency, FSEvents on macOS most visibly, can still report those setup writes once the first run is under way, which restarts it and makes the later, intentional write the second restart. The test has been marked flaky on macOS x64 for that reason. Wait for the first drain, then require that the write is followed by a restart and a drain, ignoring whatever the setup produced before it, and drop the flaky marker. Refs: nodejs#54534 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

test-wasi-pthread fails now and then on every platform with "Assertion failed: r == 0 (c/pthread.c: main: 17)", i.e. pthread_create() itself reporting an error. The fixture implements `thread-spawn` by starting a Worker and blocking in Atomics.wait(result, 0, 0, 1000) until the worker signals that it has instantiated the module. Two things go wrong there: the worker signals success by storing 0, the value the main thread is already waiting on, so when the worker is quicker than the main thread its notify is lost and the wait runs into the timeout; and one second is not always enough for a Worker to start and instantiate a threads build on the slower CI hosts (arm debug, Windows, macOS). Either way spawn() returns -6 and wasi-libc turns that into a pthread_create() failure. Wait on a sentinel value that neither outcome writes, and give the worker a platform-scaled 30 seconds. Drop the flaky markers. Fixes: nodejs#64226 Refs: nodejs#59146 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

test-dns-resolver-max-timeout times a query with `{ timeout: 500, tries: 3 }` against one that also sets `maxTimeout: 500` and asserts the first took longer. c-ares only expires a try when cares_wrap's timer fires, every `timeout` ms, so each try costs one or two ticks depending on sub-millisecond ordering, and uncapped retries also get 0.5-1x jitter. That leaves the capped run anywhere in 1500-3000 ms and the uncapped one in 3000-4500 ms; on a busy rhel10-ppc64le host both came out at 3005 ms. With `timeout: 100, tries: 5` the ranges (about 600-1000 ms and 5000-7500 ms measured under load) cannot meet. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

codebytere · 2026-09-05T18:01:07Z

@lpinca ah, missed that one, thanks - dropped it here in favor of #65770 and swapped in a fix for test-dns-resolver-max-timeout instead.

panva · 2026-09-05T18:10:57Z

i'm suspecting a number of test/parallel/test-bench-* jobs are going to pop up in the report tomorrow, how about we preemptively deflake those too before starting another full CI?

cc @jasnell

off just these two deflake PRs that ran its CI today:

Most node:bench tests record a sample with `b.start(); process.hrtime.bigint(); b.end(1)`, counting on the extra clock read to make end() see a later timestamp than start(). On a debian12-x64 CI host the monotonic clock is coarse enough that all three reads returned the same value, so end() threw ERR_INVALID_STATE ("insufficient clock precision for benchmark sample") and test-bench-harness-errors failed; every test using the idiom can fail the same way there. Add test/common/bench.js with completeSample(), which spins until process.hrtime.bigint() moves between start() and end(), and use it at all of those sites, bench-runner fixtures included. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com>

test-wasi-pthread fails now and then on every platform with "Assertion failed: r == 0 (c/pthread.c: main: 17)", i.e. pthread_create() itself reporting an error. The fixture implements `thread-spawn` by starting a Worker and blocking in Atomics.wait(result, 0, 0, 1000) until the worker signals that it has instantiated the module. Two things go wrong there: the worker signals success by storing 0, the value the main thread is already waiting on, so when the worker is quicker than the main thread its notify is lost and the wait runs into the timeout; and one second is not always enough for a Worker to start and instantiate a threads build on the slower CI hosts (arm debug, Windows, macOS). Either way spawn() returns -6 and wasi-libc turns that into a pthread_create() failure. Wait on a sentinel value that neither outcome writes, and give the worker a platform-scaled 30 seconds. Drop the flaky markers. Fixes: #64226 Refs: #59146 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Refs: #54534 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>

test-dns-resolver-max-timeout times a query with `{ timeout: 500, tries: 3 }` against one that also sets `maxTimeout: 500` and asserts the first took longer. c-ares only expires a try when cares_wrap's timer fires, every `timeout` ms, so each try costs one or two ticks depending on sub-millisecond ordering, and uncapped retries also get 0.5-1x jitter. That leaves the capped run anywhere in 1500-3000 ms and the uncapped one in 3000-4500 ms; on a busy rhel10-ppc64le host both came out at 3005 ms. With `timeout: 100, tries: 5` the ranges (about 600-1000 ms and 5000-7500 ms measured under load) cannot meet. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Fixes: #64226 Refs: #59146 Refs: #54534 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>

Most node:bench tests record a sample with `b.start(); process.hrtime.bigint(); b.end(1)`, counting on the extra clock read to make end() see a later timestamp than start(). On a debian12-x64 CI host the monotonic clock is coarse enough that all three reads returned the same value, so end() threw ERR_INVALID_STATE ("insufficient clock precision for benchmark sample") and test-bench-harness-errors failed; every test using the idiom can fail the same way there. Add test/common/bench.js with completeSample(), which spins until process.hrtime.bigint() moves between start() and end(), and use it at all of those sites, bench-runner fixtures included. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Fixes: #64226 Refs: #59146 Refs: #54534 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>

test-https-proxy-request-invalid-char-in-url is the only client-proxy test that asserts the proxy logged no socket errors at all. Once the last response has been read the client destroys its tunnel, and if the proxy is still relaying the upstream's TLS close_notify at that point the client answers with a reset, which the proxy records as ECONNRESET on the CONNECT socket. That has been failing the test on macOS even though every request was routed to the sanitized URL. Keep asserting on other errors but leave connection resets out. Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Fixes: #64226 Refs: #59146 Refs: #54534 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>

test-external-memory-reasonable-size makes a child allocate 1.2 GB of external memory so that V8's --external-memory-max-reasonable-size check fires and the process aborts. The abort raises SIGABRT with all of that memory resident, and on hosts that write core files (the SmartOS CI machines in particular) the dump takes longer than the test timeout, so the test has been timing out there since it was added. Run the child under `ulimit -c 0` on POSIX, the same way test-abort-fatal-error and common.childShouldThrowAndAbort() handle their aborting children. Refs: #65589 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Fixes: #64226 Refs: #59146 Refs: #54534 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>

test-run-watch-emit-restarted expected exactly one test:watch:restarted event, but it starts run({ watch: true }) right after writing the fixtures into the watched directory. Watch backends that deliver events with some latency, FSEvents on macOS most visibly, can still report those setup writes once the first run is under way, which restarts it and makes the later, intentional write the second restart. The test has been marked flaky on macOS x64 for that reason. Wait for the first drain, then require that the write is followed by a restart and a drain, ignoring whatever the setup produced before it, and drop the flaky marker. Refs: #54534 Signed-off-by: Shelley Vohr <shelley.vohr@gmail.com> PR-URL: #65780 Fixes: #64226 Refs: #59146 Reviewed-By: Filip Skokan <panva.ip@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gürgün Dayıoğlu <hey@gurgun.day> Reviewed-By: Moshe Atlow <moshe@atlow.co.il>