Skip to content

Commit 0234235

Browse files
joao-oliveira-softtoraduh95
authored andcommitted
test_runner: add option to forbid exclusive tests
Add --test-forbid-only and the forbidOnly run() option to detect accidentally committed exclusive tests. Reject only declarations and runOnly(true) in both isolation modes, and record a run-level failure so TODO and expected failures cannot absorb the error. Document the runtime-only detection and add regression coverage for filtering, option inheritance, both isolation modes, and disabled guards. Refs: #60917 Assisted-by: a coding agent Signed-off-by: João Victor <joao.oliveira@softtor.com.br> PR-URL: #66578 Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Chemi Atlow <chemi@atlow.co.il>
1 parent dec6580 commit 0234235

9 files changed

Lines changed: 234 additions & 0 deletions

File tree

‎doc/api/cli.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3214,6 +3214,22 @@ added: v22.8.0
32143214
Require a minimum percent of covered lines. If code coverage does not reach
32153215
the threshold specified, the process will exit with code `1`.
32163216

3217+
### `--test-forbid-only`
3218+
3219+
<!-- YAML
3220+
added: REPLACEME
3221+
-->
3222+
3223+
Fails the test run if a test or suite is declared with `only: true`, including
3224+
the `test.only()`, `it.only()`, `suite.only()`, and `describe.only()` shorthands,
3225+
or if `testContext.runOnly(true)` is called. This is useful in CI to prevent
3226+
accidentally committing exclusive tests that exclude the rest of a test suite.
3227+
3228+
This option applies with either test isolation mode and takes precedence over
3229+
`--test-only`. Name filtering, TODO markers, and expected failures do not suppress
3230+
detected violations. Only declarations evaluated during the run are checked;
3231+
code inside a skipped test callback is not evaluated.
3232+
32173233
### `--test-force-exit`
32183234

32193235
<!-- YAML
@@ -4322,6 +4338,7 @@ one is included in the list below.
43224338
* `--test-coverage-include-all`
43234339
* `--test-coverage-include`
43244340
* `--test-coverage-lines`
4341+
* `--test-forbid-only`
43254342
* `--test-global-setup`
43264343
* `--test-isolation`
43274344
* `--test-name-pattern`

‎doc/api/test.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -355,6 +355,14 @@ it.todo('should do the thing', { expectFailure: true }, () => {
355355

356356
## `only` tests
357357

358+
To prevent exclusive tests from accidentally excluding tests in CI, use
359+
[`--test-forbid-only`][] or the `forbidOnly` option of [`run()`][]. When enabled,
360+
declaring a test or suite with `only: true` or calling `testContext.runOnly(true)`
361+
fails the run. This includes the `.only()` shorthands and applies with either
362+
isolation mode, even when `--test-only` is also enabled. Name filtering, TODO
363+
markers, and expected failures cannot suppress detected violations. Declarations
364+
inside callbacks that are never executed are not checked.
365+
358366
If Node.js is started with the [`--test-only`][] command-line option, or test
359367
isolation is disabled, it is possible to skip all tests except for a selected
360368
subset by passing the `only` option to the tests that should run. When a test
@@ -1706,6 +1714,9 @@ added:
17061714
- v18.9.0
17071715
- v16.19.0
17081716
changes:
1717+
- version: REPLACEME
1718+
pr-url: https://github.com/nodejs/node/pull/66578
1719+
description: Added the `forbidOnly` option.
17091720
- version: v26.2.0
17101721
pr-url: https://github.com/nodejs/node/pull/63221
17111722
description: Added the `testTagFilters` option.
@@ -1760,6 +1771,9 @@ changes:
17601771
* `forceExit` {boolean} Configures the test runner to exit the process once
17611772
all known tests have finished executing even if the event loop would
17621773
otherwise remain active. **Default:** `false`.
1774+
* `forbidOnly` {boolean} Fail the run when a test or suite is declared with
1775+
`only: true` or `testContext.runOnly(true)` is called. See [`only` tests](#only-tests).
1776+
**Default:** the value of [`--test-forbid-only`][].
17631777
* `globPatterns` {Array} An array containing the list of glob patterns to
17641778
match test files. This option cannot be used together with `files`.
17651779
**Default:** Same as [running tests from the command line][].
@@ -5158,6 +5172,7 @@ test.describe('my suite', (suite) => {
51585172
[`--test-concurrency`]: cli.md#--test-concurrency
51595173
[`--test-coverage-exclude`]: cli.md#--test-coverage-exclude
51605174
[`--test-coverage-include`]: cli.md#--test-coverage-include
5175+
[`--test-forbid-only`]: cli.md#--test-forbid-only
51615176
[`--test-name-pattern`]: cli.md#--test-name-pattern
51625177
[`--test-only`]: cli.md#--test-only
51635178
[`--test-reporter-destination`]: cli.md#--test-reporter-destination

‎doc/node.1‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1609,6 +1609,16 @@ filtering as the rest of the report.
16091609
Require a minimum percent of covered lines. If code coverage does not reach
16101610
the threshold specified, the process will exit with code \fB1\fR.
16111611
.
1612+
.It Fl -test-forbid-only
1613+
Fails the test run if a test or suite is declared with \fBonly: true\fR, including
1614+
the \fBtest.only()\fR, \fBit.only()\fR, \fBsuite.only()\fR, and \fBdescribe.only()\fR shorthands,
1615+
or if \fBtestContext.runOnly(true)\fR is called. This is useful in CI to prevent
1616+
accidentally committing exclusive tests that exclude the rest of a test suite.
1617+
This option applies with either test isolation mode and takes precedence over
1618+
\fB--test-only\fR. Name filtering, TODO markers, and expected failures do not suppress
1619+
detected violations. Only declarations evaluated during the run are checked;
1620+
code inside a skipped test callback is not evaluated.
1621+
.
16121622
.It Fl -test-force-exit
16131623
Configures the test runner to exit the process once all known tests have
16141624
finished executing even if the event loop would otherwise remain active.
@@ -2412,6 +2422,8 @@ one is included in the list below.
24122422
.It
24132423
\fB--test-coverage-lines\fR
24142424
.It
2425+
\fB--test-forbid-only\fR
2426+
.It
24152427
\fB--test-global-setup\fR
24162428
.It
24172429
\fB--test-isolation\fR

‎lib/internal/test_runner/runner.js‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -207,6 +207,7 @@ function getRunArgs(path, { forceExit,
207207
testSkipPatterns,
208208
testTagFilterExpressions,
209209
only,
210+
forbidOnly,
210211
hasFiles,
211212
testFiles,
212213
globPatterns,
@@ -253,6 +254,9 @@ function getRunArgs(path, { forceExit,
253254
if (only === true) {
254255
ArrayPrototypePush(runArgs, '--test-only');
255256
}
257+
if (forbidOnly != null && forbidOnly !== getOptionValue('--test-forbid-only')) {
258+
ArrayPrototypePush(runArgs, forbidOnly ? '--test-forbid-only' : '--no-test-forbid-only');
259+
}
256260
if (timeout != null) {
257261
ArrayPrototypePush(runArgs, `--test-timeout=${timeout}`);
258262
}
@@ -789,6 +793,7 @@ function run(options = kEmptyObject) {
789793
globalSetupPath,
790794
only,
791795
globPatterns,
796+
forbidOnly,
792797
coverage = false,
793798
coverageIncludeAll = false,
794799
lineCoverage = 0,
@@ -821,6 +826,9 @@ function run(options = kEmptyObject) {
821826
if (only != null) {
822827
validateBoolean(only, 'options.only');
823828
}
829+
if (forbidOnly != null) {
830+
validateBoolean(forbidOnly, 'options.forbidOnly');
831+
}
824832
if (globPatterns != null) {
825833
validateArray(globPatterns, 'options.globPatterns');
826834
}
@@ -1021,6 +1029,10 @@ function run(options = kEmptyObject) {
10211029
testTagFilters,
10221030
};
10231031

1032+
if (forbidOnly != null) {
1033+
globalOptions.forbidOnly = forbidOnly;
1034+
}
1035+
10241036
if (isolation === 'none') {
10251037
if (testNamePatterns != null) {
10261038
globalOptions.testNamePatterns = testNamePatterns;
@@ -1067,6 +1079,7 @@ function run(options = kEmptyObject) {
10671079
globPatterns,
10681080
only,
10691081
forceExit,
1082+
forbidOnly,
10701083
cwd,
10711084
isolation,
10721085
argv,

‎lib/internal/test_runner/test.js‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -377,6 +377,9 @@ class TestContext {
377377
}
378378

379379
runOnly(value) {
380+
if (value && this.#test.config.forbidOnly) {
381+
failForbiddenOnly(this.#test.root, 'runOnly() is not allowed when forbidOnly is enabled');
382+
}
380383
this.#test.runOnlySubtests = !!value;
381384
}
382385

@@ -575,6 +578,14 @@ function parseExpectFailure(expectFailure) {
575578
return { __proto__: null, label: undefined, match: expectFailure };
576579
}
577580

581+
function failForbiddenOnly(root, message) {
582+
// Filtering, TODO, and expected failures must not hide forbidden exclusivity.
583+
root.harness.success = false;
584+
root.diagnostic(message);
585+
process.exitCode = kGenericUserError;
586+
throw new ERR_TEST_FAILURE(message, kTestCodeFailure);
587+
}
588+
578589
class Test extends AsyncResource {
579590
reportedType = 'test';
580591
abortController;
@@ -644,6 +655,10 @@ class Test extends AsyncResource {
644655
parent.nesting + 1;
645656
const { config, isFilteringByName, isFilteringByOnly, isFilteringByTags } = parent.root.harness;
646657

658+
if (only && config.forbidOnly) {
659+
failForbiddenOnly(parent.root, 'The only option is not allowed when forbidOnly is enabled');
660+
}
661+
647662
this.root = parent.root;
648663
this.harness = null;
649664
this.config = config;

‎lib/internal/test_runner/utils.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,7 @@ function parseCommandLine() {
270270
let destinations;
271271
let isolation;
272272
let only = getOptionValue('--test-only');
273+
const forbidOnly = getOptionValue('--test-forbid-only');
273274
let reporters;
274275
let shard;
275276
let testNamePatterns = mapPatternFlagToRegExArray('--test-name-pattern');
@@ -424,6 +425,7 @@ function parseCommandLine() {
424425
functionCoverage,
425426
lineCoverage,
426427
only,
428+
forbidOnly,
427429
reporters,
428430
setup,
429431
globalSetupPath,

‎src/node_options.cc‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1267,6 +1267,12 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
12671267
kAllowedInEnvvar,
12681268
false,
12691269
OptionNamespaces::kTestRunnerNamespace);
1270+
AddOption("--test-forbid-only",
1271+
"fail tests that use the 'only' option",
1272+
BOOL_FIELD(test_forbid_only),
1273+
kAllowedInEnvvar,
1274+
false,
1275+
OptionNamespaces::kTestRunnerNamespace);
12701276
AddOption("--test-shard",
12711277
"run test at specific shard",
12721278
&EnvironmentOptions::test_shard,

‎src/node_options.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,7 @@ class EnvironmentOptions : public Options {
274274
DEFINE_BOOL_FIELD(test_randomize) = false;
275275
DEFINE_BOOL_FIELD(has_test_random_seed) = false;
276276
DEFINE_BOOL_FIELD(test_only) = false;
277+
DEFINE_BOOL_FIELD(test_forbid_only) = false;
277278
DEFINE_BOOL_FIELD(test_udp_no_try_send) = false;
278279
DEFINE_BOOL_FIELD(coverage_include_all) = false;
279280
DEFINE_BOOL_FIELD(throw_deprecation) = false;
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
'use strict';
2+
const common = require('../common');
3+
const assert = require('node:assert');
4+
const { writeFileSync } = require('node:fs');
5+
const { test, run } = require('node:test');
6+
const tmpdir = require('../common/tmpdir');
7+
8+
tmpdir.refresh();
9+
const fixture = tmpdir.resolve('only.test.js');
10+
writeFileSync(fixture, `
11+
const { test } = require('node:test');
12+
test.only('focused test', () => {});
13+
test('ordinary test', () => {});
14+
`);
15+
const env = { ...process.env, NODE_OPTIONS: '', FORCE_COLOR: '0' };
16+
delete env.NODE_TEST_CONTEXT;
17+
const onlyIfNodeOptionsSupport = { skip: process.config.variables.node_without_node_options };
18+
19+
function createFixture(name, source) {
20+
const file = tmpdir.resolve(`${name}.js`);
21+
writeFileSync(file, `
22+
const { test, it, suite, describe } = require('node:test');
23+
${source}
24+
`);
25+
return file;
26+
}
27+
28+
function assertForbidden({ code, signal, stdout, stderr }) {
29+
assert.strictEqual(signal, null);
30+
assert.strictEqual(code, 1, stdout + stderr);
31+
assert.match(stdout + stderr, /(?:only|runOnly).*not allowed/);
32+
assert.doesNotMatch(stdout, /BODY_EXECUTED/);
33+
}
34+
35+
const cases = {
36+
'test-only': "test.only('focused test', () => { console.log('BODY_EXECUTED'); });",
37+
'it-only': "it.only('focused test', () => { console.log('BODY_EXECUTED'); });",
38+
'suite-only': "suite.only('focused suite', () => { console.log('BODY_EXECUTED'); });",
39+
'describe-only': "describe.only('focused suite', () => { console.log('BODY_EXECUTED'); });",
40+
'only-option': "test('focused test', { only: true }, () => { console.log('BODY_EXECUTED'); });",
41+
'suite-option': "suite('focused suite', { only: true }, () => { console.log('BODY_EXECUTED'); });",
42+
'subtest-option': "test('parent', async (t) => { await t.test('child', { only: true }, () => {}); });",
43+
'run-only': "test('parent', (t) => { t.runOnly(true); console.log('BODY_EXECUTED'); });",
44+
'skipped-only': "test('focused test', { only: true, skip: true }, () => {});",
45+
'todo-only': "test('focused test', { only: true, todo: true }, () => {});",
46+
'expected-failure-only': "test('focused test', { only: true, expectFailure: true }, () => {});",
47+
'todo-parent': "test.todo('parent', () => { test.only('child', () => {}); });",
48+
'expected-failure-parent': "test('parent', { expectFailure: true }, (t) => { t.runOnly(true); });",
49+
};
50+
const caseFiles = Object.entries(cases).map(([name, source]) => [name, createFixture(name, source)]);
51+
52+
for (const isolation of ['process', 'none']) {
53+
test(`run() rejects only with isolation=${isolation}`, async () => {
54+
const runner = tmpdir.resolve(`runner-${isolation}.js`);
55+
writeFileSync(runner, `
56+
const { run } = require('node:test');
57+
const { tap } = require('node:test/reporters');
58+
run({
59+
files: [${JSON.stringify(fixture)}],
60+
isolation: ${JSON.stringify(isolation)},
61+
forbidOnly: true,
62+
}).on('test:fail', () => { process.exitCode = 1; }).compose(tap).pipe(process.stdout);
63+
`);
64+
const { code, signal, stdout, stderr } = await common.spawnPromisified(process.execPath, [runner], { env });
65+
assertForbidden({ code, signal, stdout, stderr });
66+
});
67+
68+
for (const [name, file] of caseFiles) {
69+
test(`CLI rejects ${name} with isolation=${isolation}`, async () => {
70+
assertForbidden(await common.spawnPromisified(process.execPath, [
71+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap', '--test-forbid-only', file,
72+
], { env }));
73+
});
74+
}
75+
76+
for (const args of [['--test-only'], ['--test-name-pattern=ordinary'], ['--test-skip-pattern=focused']]) {
77+
test(`filter ${args[0]} cannot hide only with isolation=${isolation}`, async () => {
78+
assertForbidden(await common.spawnPromisified(process.execPath, [
79+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap', '--test-forbid-only', ...args, fixture,
80+
], { env }));
81+
});
82+
}
83+
84+
test(`NODE_OPTIONS enables forbidOnly with isolation=${isolation}`, onlyIfNodeOptionsSupport, async () => {
85+
assertForbidden(await common.spawnPromisified(process.execPath, [
86+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap', fixture,
87+
], { env: { ...env, NODE_OPTIONS: '--test-forbid-only' } }));
88+
});
89+
90+
test(`filtered suites cannot hide only with isolation=${isolation}`, async () => {
91+
const file = createFixture(`filtered-${isolation}`, `
92+
suite('filtered', () => { it.only('child', () => {}); });
93+
test('ordinary', () => {});
94+
`);
95+
assertForbidden(await common.spawnPromisified(process.execPath, [
96+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap',
97+
'--test-forbid-only', '--test-name-pattern=ordinary', file,
98+
], { env }));
99+
});
100+
101+
for (const source of ['CLI', 'NODE_OPTIONS']) {
102+
const options = source === 'NODE_OPTIONS' ? onlyIfNodeOptionsSupport : {};
103+
test(`run() can override ${source} with forbidOnly=false and isolation=${isolation}`, options, async () => {
104+
const runner = createFixture(`override-${source}-${isolation}`, `
105+
const { run } = require('node:test');
106+
const { tap } = require('node:test/reporters');
107+
run({ files: [${JSON.stringify(fixture)}], isolation: ${JSON.stringify(isolation)}, forbidOnly: false })
108+
.on('test:fail', () => { process.exitCode = 1; }).compose(tap).pipe(process.stdout);
109+
`);
110+
const args = source === 'CLI' ? ['--test-forbid-only', runner] : [runner];
111+
const childEnv = source === 'NODE_OPTIONS' ? { ...env, NODE_OPTIONS: '--test-forbid-only' } : env;
112+
const { code, signal, stdout, stderr } = await common.spawnPromisified(process.execPath, args, { env: childEnv });
113+
assert.strictEqual(signal, null);
114+
assert.strictEqual(code, 0, stdout + stderr);
115+
assert.match(stdout, /ok 1 - focused test/);
116+
});
117+
}
118+
119+
test(`ordinary tests still run with isolation=${isolation}`, async () => {
120+
const file = createFixture(`ordinary-${isolation}`, `
121+
test('ordinary', () => {});
122+
test('explicit false', { only: false }, () => {});
123+
test('runOnly false', (t) => { t.runOnly(false); });
124+
`);
125+
const { code, signal, stdout, stderr } = await common.spawnPromisified(process.execPath, [
126+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap', '--test-forbid-only', file,
127+
], { env });
128+
assert.strictEqual(signal, null);
129+
assert.strictEqual(code, 0, stdout + stderr);
130+
// Explicit only:false is filtered out in the existing none-isolation mode.
131+
assert.match(stdout, isolation === 'none' ? /# pass 2/ : /# pass 3/);
132+
});
133+
134+
for (const flags of [[], ['--test-forbid-only', '--no-test-forbid-only']]) {
135+
test(`only still filters when disabled (${flags}) with isolation=${isolation}`, async () => {
136+
const { code, signal, stdout, stderr } = await common.spawnPromisified(process.execPath, [
137+
'--test', `--test-isolation=${isolation}`, '--test-reporter=tap', '--test-only', ...flags, fixture,
138+
], { env });
139+
assert.strictEqual(signal, null);
140+
assert.strictEqual(code, 0, stdout + stderr);
141+
assert.match(stdout, /# pass 1/);
142+
assert.doesNotMatch(stdout, /ordinary test/);
143+
});
144+
}
145+
}
146+
147+
test('run() validates forbidOnly', () => {
148+
for (const forbidOnly of [0, 1, '', 'true', {}, []]) {
149+
assert.throws(() => run({ forbidOnly }), {
150+
code: 'ERR_INVALID_ARG_TYPE',
151+
});
152+
}
153+
});

0 commit comments

Comments
 (0)