From 109c7c6acc2cb53fc54e245db7f689e3d5342d43 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 3 Nov 2026 03:02:57 -0800 Subject: [PATCH 1/2] Add the Spanish-derived particles of Filipino surnames to fil.toml (strict-gap batch B) de, del, san, la and los: strict fil read each as a name word, breaking 473, 94, 70, 46 and 19 records. No default or plain-choice parse moves; under strict fil, del, la and san read as never-given particles (their given readings are es.toml's), which the reviewed ungated table records. Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 2 ++ docs/release_log.rst | 2 +- docs/usage.rst | 4 ++-- nameparser/_types.py | 2 +- nameparser/config/_index.py | 10 +++++----- nameparser/vocab/fil.toml | 20 ++++++++++++++++++++ tests/v2/pipeline/test_assemble.py | 4 ++-- tests/v2/test_parser.py | 2 +- tests/v2/test_reading_index.py | 1 + 9 files changed, 35 insertions(+), 12 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index 03f859171..be741b162 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -543,3 +543,5 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **The audit of every other `mul.toml` word, against the 2026-11-02 #1005 criterion** (incidence: tokens of every record, split at whitespace and commas, edge periods stripped, casefolded, also with every period removed; bearers by `tools/name_usage.py --no-wikidata --boxes`). `dr` and `prof` are written in 143 and 54 ORCID countries (3 records or more) and 128 and 13 PSC nationalities; `doctor`, `professor` and `docent` in English-context data from many countries (`docent` by ClinicalTrials.gov's Swedish, Finnish, Czech and Uzbek officials, 79 of its 80 names after a comma), none with a name reading in any file; the numerals and `&` everywhere (FEC `ii` 2,838, `iii` 7,872); `phd` in 75 ORCID countries; the other #1021 words as that entry measured. All stay. Recorded, not acted on: `doctor` is borne as a surname (Census 2010 3,465; PSC `Cyrus Nadir Doctor`, `Maricar Doctor`), which the trailing position reads as a name already, and `doctor` and `professor` as southern African given names (ORCID `Doctor Senamela`, `Professor Khupe`), which the title takes in front under every choice, the default included: a vocabulary-collisions question, not a `mul.toml` one. - **`ing` stays.** It is written after a name by Czech, Slovak, Romanian, Austrian, Belgian (`Gunter Reekmans, Ing`), Québécois (`ing.`) and Serbian (`Dipl Ing`) officials and as the pan-European `EUR ING`, and its name readings (Census 2010 1,814, 56% Asian or Pacific Islander; a given name in Malaysia and Singapore and in Sweden) sit in `mul.toml` with it. Moving its credential to cs, sk, de, ro, nl, fr, sr and hr and its names to en and sv was measured under the default, each tag's plain choice and each strict choice (162 configurations) over the 29,097 local-corpus names holding `ing` or `mba`: the default and every plain choice unmoved; those eight languages' strict choices moved 52 names, the credential ungated (`Alvin Ing` → suffix 'Ing'); every other strict choice moved 26, the credential lost (`Daniela Ponce, Ing` → given 'Ing'). Worse both ways. - **Recompute.** `tools/differential/config_diff.py gaps` (#1025) on the tree before and after; the `ms` split and the `ing` move as `config_diff.py diff` over the configurations named, with `--readings`, in this tree and in a scratch tree carrying the move, the two files compared line by line; the simulated trees are in no branch. +- **2026-11-03 #1005, strict-gap batch B, fil (Derek) — fil.toml TAKES THE SPANISH-DERIVED PARTICLES OF FILIPINO SURNAMES, `de`, `del`, `san`, `la` AND `los`; NO DEFAULT OR PLAIN-CHOICE PARSE MOVES; UNDER STRICT `fil`, `del`, `la` AND `san` READ AS NEVER-GIVEN.** One of the five strict-gap batches decided in chat on 2026-11-03 ("All five, here in order"), on the re-ranked report (`tools/differential/config_diff.py gaps` on master f48c88a3; the batches' own entry is the 2026-11-03 #1005, the strict-gap batches entry). fil.toml held `dela` alone of these, so strict `fil` read every other one as a name word: it broke 473 records on `de`, 94 on `del`, 70 on `san`, 46 on `la` and 19 on `los` (`Helbert De La Cruz`, `Joshua Del Prado`, `Raul San Diego`, Philippine ORCID, PatentsView and PSC records), and with batch A's `jr` the language's 1,066 broken records fall to 28 (simulated with all five batches). Each entry copies es.toml's kind and properties (`de` and `los` state `may_be_given = false`, no file holding a given reading of either). + - **What moves.** The compiled default does not (`tools/vocab/build.py` reports `_compiled.py` current). Over a grid of 21 Filipino names (each comma shape, cased, upper and lower spellings, a title and a generation beside them) under the three orders, the default and plain `fil` move no cell; strict `fil` and strict `fil` + `en` move 50 each, toward the default's reading save one class: `del`, `la` and `san` hold a `given` reading in es.toml alone, which strict `fil` drops while it keeps fil.toml's particle reading, so there they read as never-given particles, and a bare `Del Rosario` reads family 'Del Rosario' where the default reads given 'Del' and reports particle-or-given. That reads the Filipino surname as the surname it is; `_reading_index.strict_ungated`'s reviewed table gains `fil: del, la, san`, as it holds `it`'s for the same three. A plain choice reads `Reading.languages` of `de` and `la` with `fil` among the files. diff --git a/docs/release_log.rst b/docs/release_log.rst index bc05df069..2eea013a8 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -16,7 +16,7 @@ Release Log - **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704` - **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default() | Lexicon(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959` - **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754` - - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` + - **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fil', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753` - **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753` - **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753` - **Add parser_for(..., strict=True), to read only the chosen languages' words.** ``parser_for(Locale(code="ko", lexicon=Lexicon.empty(), languages=frozenset({"ko"})), strict=True).parse("Kim Do Hyun")`` gives middle ``Do``, family ``Hyun``, where the default, as in 1.4.0 through 2.8.0, and a plain ``ko`` choice give family ``Do Hyun``, ``Do`` read as the Portuguese particle. Only the chosen languages' vocabulary files are read, with the words every language writes alike (``Dr``, ``PhD``, ``Mr``, the numerals) and the words written in other scripts; every other language's words leave, the rest of the English credentials and salutations included, so ``Kim Min Jun, MD`` gives given ``MD`` unless ``en`` is chosen too. A strict call that chooses no language raises ``ValueError``. ``python -m nameparser`` gains ``--strict``, and ``--locale`` takes a language tag no pack has (``--locale ko``). See :doc:`locales`. :issue:`753`, :issue:`1005` diff --git a/docs/usage.rst b/docs/usage.rst index 41c338dbb..64514a686 100644 --- a/docs/usage.rst +++ b/docs/usage.rst @@ -1270,8 +1270,8 @@ file does hold turns that file's reading on, and names it: ... print(f"{tok.text}: {tok.reading.languages}") Dr.: ('mul',) Juan: () - de: ('es', 'fr', 'it', 'nl', 'pt') - la: ('es', 'fr', 'it') + de: ('es', 'fil', 'fr', 'it', 'nl', 'pt') + la: ('es', 'fil', 'fr', 'it') Vega: () III: ('mul',) diff --git a/nameparser/_types.py b/nameparser/_types.py index 9e4ce819c..04beb05b5 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -958,7 +958,7 @@ class Reading: source: ReadingSource #: The BCP 47 tags of the shipped vocabulary files that hold this #: reading of the word, on in the parse's own ``Lexicon`` - #: (``("es", "fr", "it", "nl", "pt")`` for the particle ``de`` by + #: (``("es", "fil", "fr", "it", "nl", "pt")`` for the particle ``de`` by #: default, ``("pt",)`` under a ``pt`` choice, which drops the #: others' readings of it; ``("mul",)`` for the title ``Dr.``), so #: a choice of languages names the files it turns on and not those diff --git a/nameparser/config/_index.py b/nameparser/config/_index.py index f90a9ccf2..63da85645 100644 --- a/nameparser/config/_index.py +++ b/nameparser/config/_index.py @@ -573,14 +573,14 @@ ('dcvo', (('en', 'credential', ()),)), ('dd', (('en', 'credential', ()),)), ('dds', (('en', 'credential', ()),)), - ('de', (('es', 'preposition', ()), ('fr', 'preposition', ()), ('it', 'preposition', ()), ('nl', 'article', ()), ('pt', 'preposition', ()))), + ('de', (('es', 'preposition', ()), ('fil', 'preposition', ()), ('fr', 'preposition', ()), ('it', 'preposition', ()), ('nl', 'article', ()), ('pt', 'preposition', ()))), ("de'", (('it', 'preposition', ()),)), ('deacon', (('en', 'religious', ()), ('en', 'given', ()), ('en', 'family', ()))), ('dean', (('en', 'office', ()), ('en', 'given', ()), ('en', 'family', ()))), ('ded', (('en', 'credential', ()),)), ('degli', (('it', 'preposition', ()),)), ('dei', (('it', 'preposition', ()),)), - ('del', (('es', 'preposition', ()), ('es', 'given', ()), ('it', 'preposition', ()))), + ('del', (('es', 'preposition', ()), ('es', 'given', ()), ('fil', 'preposition', ()), ('it', 'preposition', ()))), ('dela', (('fil', 'preposition', ()), ('fil', 'given', ()))), ('delegate', (('en', 'office', ()),)), ("dell'", (('it', 'preposition', ()),)), @@ -975,7 +975,7 @@ ('kreivi', (('fi', 'nobility', ()),)), ('kt', (('en', 'credential', (('cased', 'Kt'),)),)), ("l'", (('fr', 'article', ()), ('it', 'article', ()))), - ('la', (('es', 'article', ()), ('es', 'given', ()), ('fr', 'article', ()), ('it', 'article', ()))), + ('la', (('es', 'article', ()), ('es', 'given', ()), ('fil', 'article', ()), ('fr', 'article', ()), ('it', 'article', ()))), ('lac', (('en', 'credential', ()), ('en', 'family', ()))), ('ladc', (('en', 'credential', ()),)), ('lady', (('en', 'nobility', (('attaches', ('justice',)),)),)), @@ -1035,7 +1035,7 @@ ('lmsw', (('en', 'credential', ()),)), ('lmt', (('en', 'credential', ()),)), ('lord', (('en', 'nobility', (('attaches', ('chancellor', 'justice')),)),)), - ('los', (('es', 'article', ()),)), + ('los', (('es', 'article', ()), ('fil', 'article', ()))), ('lp', (('en', 'credential', ()),)), ('lpa', (('en', 'credential', ()),)), ('lpc', (('en', 'credential', ()),)), @@ -1531,7 +1531,7 @@ ('sainte', (('fr', 'religious', ()),)), ('same', (('en', 'credential', ()),)), ('saml', (('en', 'given', (('same_as', 'samuel'),)),)), - ('san', (('es', 'saint', ()), ('es', 'given', ()), ('it', 'saint', ()))), + ('san', (('es', 'saint', ()), ('es', 'given', ()), ('fil', 'saint', ()), ('it', 'saint', ()))), ('santa', (('es', 'saint', ()), ('es', 'given', ()), ('it', 'saint', ()), ('pt', 'saint', ()))), ('saoshyant', (('en', 'religious', ()),)), ('sasm', (('en', 'credential', ()),)), diff --git a/nameparser/vocab/fil.toml b/nameparser/vocab/fil.toml index c80829399..167b17e20 100644 --- a/nameparser/vocab/fil.toml +++ b/nameparser/vocab/fil.toml @@ -12,6 +12,14 @@ salutation = { issue = 606, gender = "f", note = "borne: ~500 surnames, in Phili text = "ginoo" salutation = { issue = 606, gender = "m", note = "borne: ~500 surnames, in Philippine lists writing Surname, Given with a comma, forebears.io, worldwide, rounded, 2026-10-04; shipped under the graf class (decisions/salutation-titles)", after_given = true } +[[word]] +text = "de" +preposition = { may_be_given = false, expands = "of: Spanish-derived Filipino surnames, \"De Guzman\", \"De La Cruz\"", note = "strict fil broke 473 records on it while only es.toml and others held it (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 } + +[[word]] +text = "del" +preposition = { expands = "of the: \"Del Rosario\"", note = "strict fil broke 94 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 } + [[word]] text = "dela" given = { us_ssa = 565, as_of = 2026-10-25, note = "US SSA births summed over both sexes; nam_dict band 1 in Germany, 1 in Bulgaria" } @@ -19,6 +27,18 @@ given = { us_ssa = 565, as_of = 2026-10-25, note = "US SSA births summed over bo expands = "of the, de la contracted: \"Dela Cruz\"" note = "Dela is a short form of Adela, and an African name meaning saviour/redeemer -- so the Filipino surname particle is not the only reading" +[[word]] +text = "la" +article = { expands = "the, fem. sg.: \"De La Cruz\"", note = "strict fil broke 46 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 } + +[[word]] +text = "los" +article = { may_be_given = false, expands = "the, masc. pl.: \"De Los Santos\"", note = "strict fil broke 19 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 } + +[[word]] +text = "san" +saint = { expands = "Saint: \"San Juan\", \"San Jose\"", note = "strict fil broke 70 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 } + [[word]] text = "jo" given = { us_ssa = 180911, wikidata = 974, as_of = 2026-10-26, same_as = "josé", note = "José written short, in Philippine usage; the counts are of Jo, a given name of its own (US SSA births summed over both sexes, F 179,057, M 1,854; Wikidata a lower bound) and a Korean surname, so a period-marked Jo. opening the given part is a given name on every reading (#789)" } diff --git a/tests/v2/pipeline/test_assemble.py b/tests/v2/pipeline/test_assemble.py index bca3b194c..fdefb5cfb 100644 --- a/tests/v2/pipeline/test_assemble.py +++ b/tests/v2/pipeline/test_assemble.py @@ -482,8 +482,8 @@ def test_every_suffix_words_reading_alternative_and_index_kind_agree( _EXPECTED_LANGUAGES = [ # a particle, by its particle rows only ('la' is a given name in es # too, a reading the parse did not take) - ("Dr. Juan de la Vega III", "de", ("es", "fr", "it", "nl", "pt")), - ("Dr. Juan de la Vega III", "la", ("es", "fr", "it")), + ("Dr. Juan de la Vega III", "de", ("es", "fil", "fr", "it", "nl", "pt")), + ("Dr. Juan de la Vega III", "la", ("es", "fil", "fr", "it")), ("Dr. Juan de la Vega III", "Dr.", ("mul",)), ("Dr. Juan de la Vega III", "III", ("mul",)), ("John Smith PhD", "PhD", ("mul",)), diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index 51ad485aa..6fa828cf4 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -2972,7 +2972,7 @@ def test_reading_languages_follow_the_choice() -> None: assert _languages_of(pt, "Vincent van Gogh", "van") == ("nl",) # with no choice, the files on by default, as before assert _languages_of(Parser(), "Ana de Souza", "de") == ( - "es", "fr", "it", "nl", "pt") + "es", "fil", "fr", "it", "nl", "pt") def test_reading_languages_under_a_strict_choice_name_the_chosen_files() -> None: diff --git a/tests/v2/test_reading_index.py b/tests/v2/test_reading_index.py index 08eb54fd6..349b73185 100644 --- a/tests/v2/test_reading_index.py +++ b/tests/v2/test_reading_index.py @@ -348,6 +348,7 @@ def test_a_file_holding_no_reading_is_chosen_only_strictly() -> None: "ar": {"cheikh", "sultan"}, "cs": {"st"}, "fr": {"la", "marquis", "marquise", "st"}, + "fil": {"del", "la", "san"}, "ga": {"mac"}, "gd": {"mac", "nic"}, "hr": {"st"}, From edf59977d2badf2c6e76cddf52b4cdc0833f7c15 Mon Sep 17 00:00:00 2001 From: Derek Gulbranson Date: Tue, 3 Nov 2026 11:58:26 -0800 Subject: [PATCH 2/2] Review fixes: the comma cost and the Philippine records stated, las noted, recompute, the table's counts dated Co-Authored-By: Claude Opus 5.5 --- docs/design/decisions/vocabulary-model.md | 5 +++-- tests/v2/test_reading_index.py | 5 +++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/design/decisions/vocabulary-model.md b/docs/design/decisions/vocabulary-model.md index be741b162..f8300cfbf 100644 --- a/docs/design/decisions/vocabulary-model.md +++ b/docs/design/decisions/vocabulary-model.md @@ -454,7 +454,7 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **2026-10-31 #753, phase 3j (its built clause that a caller's listing of a dropped word is dropped by the delta superseded by 2026-11-02 #754, the `Lexicon` surface, in effect since 4i-c, 2026-11-02 #754, sub-phase 4i-c; its head's "ONLY THE CHOSEN FILES AND NATIVE-SCRIPT VOCABULARY", item 4's "every Latin-script reading leaving" and the `UNCHOOSABLE` bullet's "a strict one drops both" superseded by 2026-11-02 #1005) — FILE-STRICT IS BUILT: `parser_for(..., strict=True)` COMPILES ONLY THE CHOSEN FILES AND NATIVE-SCRIPT VOCABULARY; A STRICT CALL CHOOSING NO LANGUAGE RAISES; NO `locales.choose`; THE WORDS STRICT UNGATES ARE DERIVED AND HELD BY A TEST; A FILE HOLDING NO READING IS A STRICT CHOICE ONLY; THE DEFAULT DOES NOT MOVE.** SUPERSEDES: in the 2026-10-30 #753, file-strict entry, its verdict's "the rest of its API is proposed, not decided, and nothing is built"; its "Proposed, not decided" bullet, every item decided as proposed save `locales.choose(*tags)`, which is left out; and the files-with-no-readings bullet's "strict `vi` needs vi.toml readings first". Carries out 3j of item 11 of the 2026-10-27 #753 entry, with the definition of the 2026-10-30 entry. The four API points below were decided on 2026-10-31; the rest of the 2026-10-30 entry's proposed bullet is built as proposed (no `Locale.strict` field; the chosen files' `default = false` readings turned on and no other file's; patches and fragments applied after the strict compile, untouched; `Lexicon.given_names` and `Lexicon.surnames` outside it; packs stacked as a union; a caller's listing of a dropped word dropped by the delta). Built on master `8438ccbb`. 1. **A strict call that chooses no language raises `ValueError`.** The message names the packs given and offers a pack that chooses a language, the first pack's code where a strict choice accepts it as one (`parser_for(locales.ZH, strict=True)` offers `Locale(code='zh', lexicon=Lexicon.empty(), languages=frozenset({'zh'}))`), else `en`; tests/v2/test_parser.py runs the offered code. 2. **No public `locales.choose`.** A bare choice is a `Locale` with an empty lexicon and the tags, the spelling docs/locales.rst teaches. - 3. **The words strict ungates are derived**, `_reading_index.strict_ungated`, each strict choice against the plain choice of the same tags and never against the default's markers, which miss a reading the choice turns on; tests/v2/test_reading_index.py holds every one-tag strict choice to a per-tag table, which is the 2026-10-30 entry's 16 words and 27 word-and-tag pairs over 13 tags exactly, so a word that joins or leaves is a reviewed diff. + 3. **The words strict ungates are derived**, `_reading_index.strict_ungated`, each strict choice against the plain choice of the same tags and never against the default's markers, which miss a reading the choice turns on; tests/v2/test_reading_index.py holds every one-tag strict choice to a per-tag table, which was the 2026-10-30 entry's 16 words and 27 word-and-tag pairs over 13 tags exactly on 2026-10-31, the strict-gap batches of 2026-11-03 moving it, so a word that joins or leaves is a reviewed diff. 4. **A strict choice accepts a tag whose file holds no reading** (hu, kk, kok, vi and zu today): under strict it is not inert, every Latin-script reading leaving. A plain choice of one is still refused, its message now saying a strict one accepts it. The reading index records every file's tag for this (`FILES` in nameparser/config/_index.py, read by `_reading_index.files`), a file holding only exclusions writing no row of `READINGS`. - The build refuses `prefer` and `strict` as data, saying a strength is the caller's (`parser_for(..., strict=True)`), where it had said each was reserved for phase 3; the RFC's phase 3 API section had decided the message. - `_reading_index`'s `UNCHOOSABLE` comment and `selected`'s refusal text, which said no choice removes a reading of `mul` or `und`, now say a plain choice never does and a strict one drops both. @@ -544,4 +544,5 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names) - **`ing` stays.** It is written after a name by Czech, Slovak, Romanian, Austrian, Belgian (`Gunter Reekmans, Ing`), Québécois (`ing.`) and Serbian (`Dipl Ing`) officials and as the pan-European `EUR ING`, and its name readings (Census 2010 1,814, 56% Asian or Pacific Islander; a given name in Malaysia and Singapore and in Sweden) sit in `mul.toml` with it. Moving its credential to cs, sk, de, ro, nl, fr, sr and hr and its names to en and sv was measured under the default, each tag's plain choice and each strict choice (162 configurations) over the 29,097 local-corpus names holding `ing` or `mba`: the default and every plain choice unmoved; those eight languages' strict choices moved 52 names, the credential ungated (`Alvin Ing` → suffix 'Ing'); every other strict choice moved 26, the credential lost (`Daniela Ponce, Ing` → given 'Ing'). Worse both ways. - **Recompute.** `tools/differential/config_diff.py gaps` (#1025) on the tree before and after; the `ms` split and the `ing` move as `config_diff.py diff` over the configurations named, with `--readings`, in this tree and in a scratch tree carrying the move, the two files compared line by line; the simulated trees are in no branch. - **2026-11-03 #1005, strict-gap batch B, fil (Derek) — fil.toml TAKES THE SPANISH-DERIVED PARTICLES OF FILIPINO SURNAMES, `de`, `del`, `san`, `la` AND `los`; NO DEFAULT OR PLAIN-CHOICE PARSE MOVES; UNDER STRICT `fil`, `del`, `la` AND `san` READ AS NEVER-GIVEN.** One of the five strict-gap batches decided in chat on 2026-11-03 ("All five, here in order"), on the re-ranked report (`tools/differential/config_diff.py gaps` on master f48c88a3; the batches' own entry is the 2026-11-03 #1005, the strict-gap batches entry). fil.toml held `dela` alone of these, so strict `fil` read every other one as a name word: it broke 473 records on `de`, 94 on `del`, 70 on `san`, 46 on `la` and 19 on `los` (`Helbert De La Cruz`, `Joshua Del Prado`, `Raul San Diego`, Philippine ORCID, PatentsView and PSC records), and with batch A's `jr` the language's 1,066 broken records fall to 28 (simulated with all five batches). Each entry copies es.toml's kind and properties (`de` and `los` state `may_be_given = false`, no file holding a given reading of either). - - **What moves.** The compiled default does not (`tools/vocab/build.py` reports `_compiled.py` current). Over a grid of 21 Filipino names (each comma shape, cased, upper and lower spellings, a title and a generation beside them) under the three orders, the default and plain `fil` move no cell; strict `fil` and strict `fil` + `en` move 50 each, toward the default's reading save one class: `del`, `la` and `san` hold a `given` reading in es.toml alone, which strict `fil` drops while it keeps fil.toml's particle reading, so there they read as never-given particles, and a bare `Del Rosario` reads family 'Del Rosario' where the default reads given 'Del' and reports particle-or-given. That reads the Filipino surname as the surname it is; `_reading_index.strict_ungated`'s reviewed table gains `fil: del, la, san`, as it holds `it`'s for the same three. A plain choice reads `Reading.languages` of `de` and `la` with `fil` among the files. + - **What moves.** The compiled default does not (`tools/vocab/build.py` reports `_compiled.py` current). Over a grid of 21 Filipino names (each comma shape, cased, upper and lower spellings, a title and a generation beside them) under the three orders, the default and plain `fil` move no cell; strict `fil` and strict `fil` + `en` move 50 each, toward the default's reading save one class: `del`, `la` and `san` hold a `given` reading in es.toml alone, which strict `fil` drops while it keeps fil.toml's particle reading, so there they read as never-given particles, and a bare `Del Rosario` reads family 'Del Rosario' where the default reads given 'Del' and reports particle-or-given. That reads the Filipino surname as the surname it is; `_reading_index.strict_ungated`'s reviewed table gains `fil: del, la, san`, as it holds `it`'s for the same three. The same class after a family comma loses the given name: `Santos, Del` reads family 'Santos Del' where the default and plain `fil` read given 'Del', as the default already reads `Santos, De`; 26 names of a 1-in-4 sample of the local corpora take that shape (`BITTNER, DEL`, mostly FEC). On Philippine records the class costs 2 (`La Reina Eunice Santos` and `La Trang` read all-family, `la` broke 2 under strict `fil` on the change), `del` and `san` none (the review's run, 2026-11-03). Not added: `las` (`Las Piñas`, 3 Philippine records under strict `fil`), left for a later batch. A plain choice reads `Reading.languages` of `de` and `la` with `fil` among the files (the default too; plain `fil` names `fil` alone). + - **Recompute.** `tools/differential/config_diff.py gaps fil` on the tree before and after; the grid is the 21 names `Juan De La Cruz`, `Juan dela Cruz`, `Maria Del Rosario`, `Jose San Juan`, `Juan De Los Santos`, `De La Cruz, Juan`, `Del Rosario, Maria Luisa`, `, De La Cruz`, `Cruz, Juan de la`, `JUAN DE LA CRUZ`, `juan de la cruz`, `Juan De Guzman Jr.`, `Del Rosario`, `La Cruz`, `San Juan`, `Juan Del`, `Maria La Paz Reyes`, `Dr. Jose De Leon`, `Del Pilar Santos`, `Juan Santos, De La` and `Ma. Teresa De Los Reyes`, each parsed in both trees under the three orders and `default`, `plain:fil`, `strict:fil` and `strict:fil+en` (`config_diff.build` names each), and the cells compared; the review's own grid of 39 names agreed in direction, and `config_diff.py diff default plain:fil --local all --match '(?i)\b(de|del|la|san|los)\b' --sample 4` read 0 of 61,947 names differently in either tree. diff --git a/tests/v2/test_reading_index.py b/tests/v2/test_reading_index.py index 349b73185..c05d0289a 100644 --- a/tests/v2/test_reading_index.py +++ b/tests/v2/test_reading_index.py @@ -340,15 +340,16 @@ def test_a_file_holding_no_reading_is_chosen_only_strictly() -> None: #: The words each strict choice of one tag ungates against the plain #: choice of the same tag (decisions/vocabulary-model's 2026-10-30 #753, -#: file-strict entry: 16 words, 27 word-and-tag pairs over 13 tags). A +#: file-strict entry measured 16 words, 27 word-and-tag pairs over 13 +#: tags on 2026-10-31; the strict-gap batches of 2026-11-03 moved it). A #: word leaves this table when a name reading is written in the chosen #: file whose names carry it, and joins it when a chosen file's #: vocabulary reading loses its gate: review each move. STRICT_UNGATED = { "ar": {"cheikh", "sultan"}, "cs": {"st"}, - "fr": {"la", "marquis", "marquise", "st"}, "fil": {"del", "la", "san"}, + "fr": {"la", "marquis", "marquise", "st"}, "ga": {"mac"}, "gd": {"mac", "nic"}, "hr": {"st"},