Text copied from web pages, PDFs, or Word documents frequently carries invisible Unicode bidirectional control characters — most commonly the right-to-left mark U+200F around RTL names. The parser doesn't strip them, so they end up embedded in the parsed attributes:
>>> from nameparser import HumanName
>>> hn = HumanName("\u200fمحمد بن سلمان\u200f")
>>> hn.first
'\u200fمحمد'
>>> hn.first == "محمد"
False(The \u200f escapes are written out here because the character is invisible; real-world input contains the raw character.)
Because the characters are invisible, the corruption is silent: values print normally but fail equality comparisons, lookups, and deduplication downstream. Constants lookups can also miss (a title or prefix with a mark glued to it no longer matches its set entry).
Suggested fix: strip bidi control characters during pre-processing, exactly as emoji are already handled (#58, re_emoji in nameparser/config/regexes.py). The relevant characters: U+200E (LRM), U+200F (RLM), U+061C (ALM), U+202A–U+202E (embedding/override), U+2066–U+2069 (isolates).
Observed on v1.3.0.
Text copied from web pages, PDFs, or Word documents frequently carries invisible Unicode bidirectional control characters — most commonly the right-to-left mark U+200F around RTL names. The parser doesn't strip them, so they end up embedded in the parsed attributes:
(The
\u200fescapes are written out here because the character is invisible; real-world input contains the raw character.)Because the characters are invisible, the corruption is silent: values print normally but fail equality comparisons, lookups, and deduplication downstream. Constants lookups can also miss (a title or prefix with a mark glued to it no longer matches its set entry).
Suggested fix: strip bidi control characters during pre-processing, exactly as emoji are already handled (#58,
re_emojiinnameparser/config/regexes.py). The relevant characters: U+200E (LRM), U+200F (RLM), U+061C (ALM), U+202A–U+202E (embedding/override), U+2066–U+2069 (isolates).Observed on v1.3.0.