Skip to content

gh-62432: unittest runner: Exit code 5 if no tests were run - #102051

Merged
gpshead merged 13 commits into
python:mainfrom
stefanor:fail-zero-tests
Apr 27, 2023
Merged

gh-62432: unittest runner: Exit code 5 if no tests were run#102051
gpshead merged 13 commits into
python:mainfrom
stefanor:fail-zero-tests

Conversation

@stefanor

@stefanor stefanor commented Feb 19, 2023 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Contributor

As discussed in https://discuss.python.org/t/unittest-fail-if-zero-tests-were-discovered/21498/7

It is common for test runner misconfiguration to fail to find any tests,
this should be an error.

Fixes: #62432

@bedevere-bot bedevere-bot mentioned this pull request Feb 19, 2023
Closed
@terryjreedy terryjreedy added type-feature A feature request or enhancement tests Tests in the Lib/test dir labels Feb 20, 2023
@terryjreedy

terryjreedy commented Feb 20, 2023

Copy link
Copy Markdown
Member

Tests/macOS is routinely failing; ignore currently.

This is a design bug fix, hence enhancement (feature change), as much as I would like to pretend otherwise.

hugovk
hugovk reviewed Feb 20, 2023
View reviewed changes
Comment thread Lib/unittest/main.py Outdated
Comment thread Lib/unittest/runner.py Outdated
Comment thread Lib/test/test_unittest/test_program.py Outdated
Comment thread Lib/test/test_unittest/test_runner.py Outdated
Comment thread Misc/NEWS.d/next/Library/2023-02-19-12-37-08.gh-issue-62432.GnBFIB.rst Outdated
Comment thread Lib/test/test_unittest/test_program.py Outdated
@stefanor
stefanor requested review from hugovk and removed request for ezio-melotti and gpshead February 20, 2023 11:51
@stefanor

stefanor commented Feb 20, 2023

Copy link
Copy Markdown
Contributor Author

@stefanor stefanor requested review from hugovk and removed request for gpshead and ezio-melotti

Urgh, that removal wasn't intentional, I just hit the re-request review button.

@gpshead gpshead self-assigned this Feb 20, 2023
@stefanor

stefanor commented Apr 26, 2023

Copy link
Copy Markdown
Contributor Author

@gpshead: Ping? (it's been 2 months)

@gpshead
gpshead enabled auto-merge (squash) April 27, 2023 01:04
@gpshead
gpshead merged commit 76632b8 into python:main Apr 27, 2023
@stefanor
stefanor deleted the fail-zero-tests branch April 27, 2023 11:49
@stefanor

stefanor commented Apr 27, 2023

Copy link
Copy Markdown
Contributor Author

Thank you!

@gpshead gpshead added the 3.12 only security fixes label Jun 13, 2023
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
b274e14 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
@copybara-service copybara-service Bot mentioned this pull request Sep 14, 2023
Merged
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
16bcbeb 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
4d8eba2 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
6c95d4d 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran AND:
9ff9bfa 
- Either test filtering is used
- Sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
8d1a40c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 14, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
bbd341c 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
a4db4a8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
14f99e8 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
609aa55 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565190992
copybara-service Bot pushed a commit to abseil/abseil-py that referenced this pull request Sep 15, 2023
@yilei @copybara-github
absltest: do not fail tests on Python 3.12+ when no tests ran and:
1edf2ab 
- Either test filtering is used
- Or sharding is used and the shard index > 0, i.e. only the first shard will fail when no tests ran on Python 3.12+.

Context: Python 3.12 unittest will now fail when no tests ran after the change from python/cpython#102051. Since `absltest` is built on top of `unittest`, it will follow this behavior change in Python 3.12.

However, when test filtering is used in `absltest`, often used via `bazel test --test_filter=<my_filter>`, the current user expectation is the `bazel test` command should NOT fail is at least one test ran. Since the test runner here has no visibility of the overall `bazel` invocation, we'll make the test not fail when test filtering is used via bazel's environment variable. This is the existing behavior before Python 3.12.

Also test absl-py on Python 3.12.

PiperOrigin-RevId: 565771935
@zware zware mentioned this pull request Jan 3, 2024
Closed
@vstinner

vstinner commented Jan 8, 2024

Copy link
Copy Markdown
Member

The new Python 3.12 message surprises users when all tests are skipped: see issue #113661.

@fishy

fishy commented Jan 8, 2024

Copy link
Copy Markdown

The PR description says:

It is common for test runner misconfiguration to fail to find any tests

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

@stefanor

stefanor commented Jan 9, 2024

Copy link
Copy Markdown
Contributor Author

Which seems to suggest that it should not fail test if all tests are skipped, as skipped tests are still "found", just intentionally skipped from running.

Yeah, I'd agree with that.

@cottsay cottsay mentioned this pull request Nov 25, 2024
Open
texhnolyze added a commit to bit-bots/bitbots_main that referenced this pull request Dec 12, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
ef65537 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

See: colcon/colcon-core#678
See: python/cpython#102051
texhnolyze added a commit to bit-bots/udp_bridge that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
d74e356 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

Additionally removed old ros 1 integration tests.

See: colcon/colcon-core#678
See: python/cpython#102051
@texhnolyze texhnolyze mentioned this pull request Dec 16, 2024
Merged
3 of 6 tasks
texhnolyze added a commit to bit-bots/dynamic_stack_decider that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
d42a46a 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

See: colcon/colcon-core#678
See: python/cpython#102051
@texhnolyze texhnolyze mentioned this pull request Dec 16, 2024
Merged
3 of 6 tasks
texhnolyze added a commit to bit-bots/udp_bridge that referenced this pull request Dec 16, 2024
@texhnolyze
test(colcon): configure our pkgs to use pytest
87adf72 
to ensure that the CI `colcon test` run works, because with a change to
python 3.12 the `unittest` standard library used by default with colcon
now exits with an error code of 5 for an empty test suite.

Additionally removed old ros 1 integration tests.

See: colcon/colcon-core#678
See: python/cpython#102051
mrg123 added a commit to mrg123/nanojev-mlx that referenced this pull request Sep 19, 2026
@mrg123
Harden CI against a silent zero-test pass
ae1ea82 
Python 3.10 is in the test matrix, and on 3.10 unittest still exits 0 when it
discovers no tests at all -- exit code 5 for "NO TESTS RAN" only landed in 3.12
(python/cpython#102051). So if tests/ were ever renamed or moved, the 3.10 leg
would report success while running nothing at all.

Capture the runner output and fail explicitly when zero tests were collected.
Once tests are actually discovered the new branch cannot trigger.

Verified both paths locally under `bash -eo pipefail`, the shell GitHub Actions
uses: a simulated zero-test run (exit 0, "Ran 0 tests") is rejected with exit 1,
and a normal 19-test run passes through untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gpshead gpshead gpshead approved these changes
@hugovk hugovk Awaiting requested review from hugovk
@ezio-melotti ezio-melotti Awaiting requested review from ezio-melotti

Assignees

@gpshead gpshead

Labels

3.12 only security fixes tests Tests in the Lib/test dir type-feature A feature request or enhancement

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

running a suite with no tests is not an error

9 participants

@stefanor @terryjreedy @vstinner @fishy @gpshead @hugovk @bedevere-bot @hauntsaninja @AlexWaygood