Skip to content

Replace invalid UTF-8 in bytes formatted with %s - #1644

Open
0z5a wants to merge 1 commit into
cel-expr:masterfrom
0z5a:fix/format-bytes-invalid-utf8
Open

0z5a wants to merge 1 commit into
cel-expr:masterfrom
0z5a:fix/format-bytes-invalid-utf8

Conversation

@0z5a

@0z5a 0z5a commented Nov 14, 2026

Copy link
Copy Markdown

The strings extension spec says a bytes argument to %s is "formatted as if string(value) was performed and any invalid UTF-8 sequences are replaced with �", with adjacent invalid sequences collapsed into one �. appendingFormatterV2 appended the raw bytes, so the result could be a CEL string that is not valid UTF-8 (details in the issue).

This uses strings.ToValidUTF8(string(argBytes), "�"), which replaces each run of invalid bytes with a single U+FFFD. Bytes nested in lists and maps go through the same code. The legacy formatter used before strings version 4 is unchanged.

Fixes #1643

Three cases are added to TestStringFormatV2: the two vectors from cel-expr/cel-spec#473 (b"\xF0abc\x8C\xF0xyz" and b"\xF0\x8C\xF0") and bytes inside a list.

Local checks with Go 1.27.1 on macOS arm64, master @ 0ef120e (Bazel is not installed here, so I used go test):

Check master This PR
go test ./ext/ -run TestStringFormatV2 with the new cases FAIL (3 new cases) ok
go test ./... ok (18 packages)
go vet ./ext/, gofmt -l on the changed files clean

This change was drafted with Claude Code (claude-opus-5-5); the test results above are from the runs described.

The CEL strings extension spec says that a bytes argument to the %s
clause of string.format is "formatted as if string(value) was
performed and any invalid UTF-8 sequences are replaced with \ufffd",
with multiple adjacent invalid sequences collapsed into a single
\ufffd.

The v2 formatter appended the raw bytes, so "%s".format([b"\xff"])
produced a CEL string that is not valid UTF-8 (string(b"\xff") is an
error), and functions such as size() then counted each stray byte as
a code point. Use strings.ToValidUTF8, which replaces each run of
invalid bytes with one replacement character, matching the spec. The
change also covers bytes nested in lists and maps, and leaves the
legacy (pre-v4) formatter unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

string.format: %s on bytes with invalid UTF-8 produces a string that is not valid UTF-8

1 participant