Skip to content

Add detail to traced values where str() is ambiguous - #729

Open
RonnyPfannschmidt wants to merge 7 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:trace-value-heuristic
Open

Add detail to traced values where str() is ambiguous#729
RonnyPfannschmidt wants to merge 7 commits into
pytest-dev:mainfrom
RonnyPfannschmidt:trace-value-heuristic

Conversation

@RonnyPfannschmidt

@RonnyPfannschmidt RonnyPfannschmidt commented Sep 12, 2026

Copy link
Copy Markdown
Member

AI-authored. I asked Claude Code (Opus 5) to work out a rendering for traced
values that keeps pytest's trace readable while surfacing the details str()
drops. The code, the tests, the measurements and the text below are the agent's
work. I read it and I am posting it, and I will follow up on review comments
myself.

Stacked on #728 — that must land first. Until it does, the diff here includes its
two commits; the change proposed by this PR is the last commit,
Add detail to traced values where str() is ambiguous.

Why

#728 fixes the two tracing crashes without touching the output, and says explicitly that
the type-visibility idea from #627/#681 should be argued on its own. This is that
argument.

Rendering every value with repr(), as #681 proposed, changes 216 of 1329 lines in a
real pytest --debug run, and the overwhelming majority of that is quotes added around
strings that were already readable. But str() does drop things a reader needs:

in the trace today the problem
left: / right: an empty string is indistinguishable from no value at all — in a comparison trace
val: with space no visible boundaries; leading and trailing space are invisible
exitstatus: 1 an IntEnum prints as a bare number, the member name is lost
collection_path: /x
path: /x
a PosixPath and a py.path.local pointing at the same place look identical
a multi-line value runs out to column 0 and reads as a trace line of its own

The rule

  • a string that reads unambiguously as itself — non-empty, printable, no spaces — stays bare
  • a string that is empty, or carries whitespace, gets repr()
  • an enum.Enum or an os.PathLike gets repr()
  • a multi-line value is drawn as a block
  • everything else keeps str(), unchanged

What it costs

Measured on a real pytest --debug run: 45 of 1329 lines change (3.4%), against 216
for blanket repr(). By category: 21 paths, 11 whitespace strings, 3 empty strings, 2
enums, 2 blocks.

  exitstatus: 1                      →  exitstatus: <ExitCode.TESTS_FAILED: 1>
  left:                              →  left: ''
  right:                             →  right: ''
  collection_path: DIR/.benchmarks   →  collection_path: PosixPath('DIR/.benchmarks')
  path: DIR/.benchmarks              →  path: local('DIR/.benchmarks')
  val: with space                    →  val: 'with space'
  plugin_name: lfplugin              →  plugin_name: lfplugin        (unchanged)
  config: <_pytest.config.Config…>   →  config: <_pytest.config.Config…>   (unchanged)

Multi-line values

Today a multi-line value breaks the layout — the continuation escapes to column 0 and
reads as a top-level trace line:

            orig: isinstance(f, pathlib.Path)
            expl: True
 +  where True = isinstance(PosixPath('/tmp/pytest-N/test_path0'), <class 'pathlib.Path'>)
 +    where <class 'pathlib.Path'> = pathlib.Path
        finish pytest_assertion_pass --> [] [hook]

With this change the value is boxed, so its extent is visible at a glance:

            orig: 'isinstance(f, pathlib.Path)'
            expl:
              | True
              |  +  where True = isinstance(PosixPath('/tmp/pytest-N/test_path0'), <class 'pathlib.Path'>)
              \  +    where <class 'pathlib.Path'> = pathlib.Path
        finish pytest_assertion_pass --> [] [hook]

Rough edges, for review

  • nodeid: test_mix.py::test_p[with space] gains quotes, because the parameter id
    contains a space. The rule fires correctly, but on something that is not really "a
    string with whitespace" in spirit.
  • orig: n or n == "" becomes orig: 'n or n == ""' — nested quotes on source text.
  • Lone surrogates now render '\ud800' rather than bare, since they are not
    isprintable(). Arguably clearer; it does change two tests added in Make hook tracing unable to fail a hook call #728.
  • Rendering a traced value costs ~0.8 µs rather than ~0.2 µs, only when tracing is
    enabled — around 0.5 ms across a whole pytest run.

Testing

  • uv run pytest — 187 passed.
  • uv run pre-commit run -a — all hooks pass.
  • _tracing.py at 100% statement and branch coverage.
  • 8 new tests: 6 covering each branch of the rule, and 2 for the repr guards this change
    newly reaches — a path-like with a broken repr, and KeyboardInterrupt raised from a
    repr.
  • pytest --debug output diffed against main: 45 changed lines, each one listed above.

🤖 Generated with Claude Code

@RonnyPfannschmidt
RonnyPfannschmidt force-pushed the trace-value-heuristic branch 2 times, most recently from 41acb1b to 566041a Compare September 12, 2026 20:51
RonnyPfannschmidt and others added 7 commits September 12, 2026 23:47
Tracing could turn a working hook call into a failing one in two ways:
an object whose __repr__/__str__ raises propagated that exception out of
the hook call (pytest-dev#424), and a lone surrogate in a hook argument or return
value produced a message the writer could not encode (pytest-dev#681).

Both are now handled in one place. _safe_repr()/_safe_str() wrap the
conversion the way pytest's saferepr does -- KeyboardInterrupt and
SystemExit still propagate, anything else is rendered as an
unpresentable-object marker -- and escape lone surrogates with
backslashreplace afterwards, which also covers surrogates that come out
of an object's own __repr__.

Traced values -- hook kwargs and the hook result -- now use repr() so
their type is visible in the log; structural labels such as the hook
name and the finish/--> markers keep using str() and stay unquoted.

Supersedes pytest-dev#627, pytest-dev#666, pytest-dev#684 and pytest-dev#716.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The previous commit also switched traced values from str() to repr(),
following the design distilled in pytest-dev#681. That is a user visible change to
pytest's --debug output, and it makes that output worse to read.

The worst case is a value that is meant to be read as a block. With
enable_assertion_pass_hook, pytest passes the assertion explanation to
pytest_assertion_pass as a multi line string. Under str() the trace
shows it as written:

    expl: {'x': [0, 1, ...} == {'x': [0, 1, ...}

      Omitting 2 identical items, use -vv to show
      Use -v to get more diff

Under repr() the same value becomes one escaped line:

    expl: "{'x': [0, 1, ...} == {'x': [0, 1, ...}\n  \n  Omitting 2 identical items, use -vv to show\n  Use -v to get more diff"

The rest is quieter but hits every run: of 439 traced kwarg values in a
real pytest --debug run, 123 render differently, and 115 of those are
nothing but quotes added around strings that were already readable --
every plugin registration line turns plugin_name: lfplugin into a
quoted string. 105 of the 674 lines in the sampled trace change, so
anything parsing that output breaks as well.

The trace is pytest UX. A fix for a crash that nobody hits in normal use
must not degrade the daily reading experience of everyone who does not
hit it. The cases where repr() genuinely helps are real -- PosixPath vs
py.path.local for two arguments that print the same path, ExitCode vs a
bare int -- but they are 15 lines out of 674, and they do not pay for
the other 115 plus the escaped blocks.

The crash fixes never depended on repr(): _safe_str() guards the
conversion and escapes lone surrogates just as well, so pytest-dev#424 and pytest-dev#681
stay fixed while pytest --debug output is byte for byte what it was
before (verified: 674 trace lines, 0 differences). The type visibility
idea is not rejected, only unbundled -- it can be argued on its own in
pytest-dev#681, as a deliberate output change with its own changelog entry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
The guards around str() were only exercised for the simple case of a
broken __str__. The exception explaining that failure can be just as
broken, and Ctrl-C has to stay reliable at both levels, so cover: an
exception whose repr fails, an exception whose repr and str both fail,
and KeyboardInterrupt raised from the value and from the explanation.

_tracing.py is at 100% statement and branch coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
BrokenStr's __repr__ was never called -- _safe_str reaches for __str__,
and the failure message is built from the type name -- so it only showed
up as an uncovered line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Trace output stays str() based, because that is what makes it readable,
but str() hides things a reader needs often enough to be worth fixing
case by case:

- an empty string is indistinguishable from no value at all, which is
  exactly wrong for a comparison trace showing left and right
- a string carrying whitespace has no visible boundaries
- an IntEnum prints as a bare number, losing the member name
- a path prints as text, so a PosixPath and a py.path.local argument
  pointing at the same place look identical
- a multi line value runs into column 0 and reads as a trace line of its
  own rather than as the value of its key

So values that read unambiguously as themselves -- a non empty printable
string without spaces, and every type whose repr adds nothing -- stay
bare, and the rest gain quotes, their type, or a block.

Multi line values are drawn as a box, each line prefixed with | and the
last with \\, so the extent of the value is visible at a glance.

Measured on a real pytest --debug run, this changes 45 of 1329 trace
lines, against 216 for rendering every value with repr(). Every changed
line carries information the previous rendering dropped.

Builds on pytest-dev#728, which must land first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
_safe_repr is only reached for enums, paths and quoted strings, which
all have working reprs in the existing tests, so its guards were dead in
coverage. A path-like with a broken repr is the pytest-dev#424 scenario applied to
a value the heuristic sends through repr.

_tracing.py is back at 100% statement and branch coverage.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant