Allow arbitrary Unicode whitespace #484

Activity

1. TristonianJones commented on Oct 23, 2025

   

   TristonianJones

   on Oct 23, 2025

   Collaborator

   More actions

   @hudlow Is this in the parser grammar? Sorry, it's been a while since I looked at this.
2. hudlow commented on Nov 6, 2025

   

   hudlow

   on Nov 6, 2025

   Last edited by hudlow

   ContributorAuthor

   More actions

   @TristonianJones Yeah, the lexis says:

   WHITESPACE ::= [\t\n\f\r ]+
   

   I actually ran into a scenario in the browser where U+00A0 (non-breaking space) was produced and wouldn't parse in cel-es (which follows the grammar pretty precisely).

   However, doing a little more research, it seems like the logical definition of whitespace in the grammar is the purpose-made Pattern_White_Space property in Unicode: https://www.unicode.org/reports/tr31/

   This is what Rust defers to: https://doc.rust-lang.org/reference/whitespace.html

   Unfortunately, this wouldn't have resolved what I encountered because non-breaking spaces are not in Pattern_White_Space. And although UTR31 does note:

   Similarly, each programming language can define its own whitespace characters or syntax characters relative to the Unicode Pattern_White_Space or Pattern_Syntax characters, with some specified set of additions or subtractions.

   ...I am a little wary of Chesterton's fence.

   I'm happy to make a PR to the grammar if we agree on the right whitespace character set. I mainly raised this because it is a frustrating sharp edge when a program appears perfectly valid but isn't. (And I anticipate the use of CEL by relatively unsophisticated users whose workflows might introduce non-ASCII whitespace but will be poorly-equipped to debug the result of resulting errors.)

   Also, I really created this issue as a record of a potential gap — this is NOT urgent.