Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/design/decisions/vocabulary-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -549,3 +549,6 @@ Open: #913 (the title words phase 2j left, read as titles though borne as names)
- **2026-11-03 #1005, strict-gap batch B, fil (Derek) — fil.toml TAKES THE SPANISH-DERIVED PARTICLES OF FILIPINO SURNAMES, `de`, `del`, `san`, `la` AND `los`; NO DEFAULT OR PLAIN-CHOICE PARSE MOVES; UNDER STRICT `fil`, `del`, `la` AND `san` READ AS NEVER-GIVEN.** One of the five strict-gap batches decided in chat on 2026-11-03 ("All five, here in order"), on the re-ranked report (`tools/differential/config_diff.py gaps` on master f48c88a3; the batches' own entry is the 2026-11-03 #1005, the strict-gap batches entry). fil.toml held `dela` alone of these, so strict `fil` read every other one as a name word: it broke 473 records on `de`, 94 on `del`, 70 on `san`, 46 on `la` and 19 on `los` (`Helbert De La Cruz`, `Joshua Del Prado`, `Raul San Diego`, Philippine ORCID, PatentsView and PSC records), and with batch A's `jr` the language's 1,066 broken records fall to 28 (simulated with all five batches). Each entry copies es.toml's kind and properties (`de` and `los` state `may_be_given = false`, no file holding a given reading of either).
- **What moves.** The compiled default does not (`tools/vocab/build.py` reports `_compiled.py` current). Over a grid of 21 Filipino names (each comma shape, cased, upper and lower spellings, a title and a generation beside them) under the three orders, the default and plain `fil` move no cell; strict `fil` and strict `fil` + `en` move 50 each, toward the default's reading save one class: `del`, `la` and `san` hold a `given` reading in es.toml alone, which strict `fil` drops while it keeps fil.toml's particle reading, so there they read as never-given particles, and a bare `Del Rosario` reads family 'Del Rosario' where the default reads given 'Del' and reports particle-or-given. That reads the Filipino surname as the surname it is; `_reading_index.strict_ungated`'s reviewed table gains `fil: del, la, san`, as it holds `it`'s for the same three. The same class after a family comma loses the given name: `Santos, Del` reads family 'Santos Del' where the default and plain `fil` read given 'Del', as the default already reads `Santos, De`; 26 names of a 1-in-4 sample of the local corpora take that shape (`BITTNER, DEL`, mostly FEC). On Philippine records the class costs 2 (`La Reina Eunice Santos` and `La Trang` read all-family, `la` broke 2 under strict `fil` on the change), `del` and `san` none (the review's run, 2026-11-03). Not added: `las` (`Las Piñas`, 3 Philippine records under strict `fil`), left for a later batch. A plain choice reads `Reading.languages` of `de` and `la` with `fil` among the files (the default too; plain `fil` names `fil` alone).
- **Recompute.** `tools/differential/config_diff.py gaps fil` on the tree before and after; the grid is the 21 names `Juan De La Cruz`, `Juan dela Cruz`, `Maria Del Rosario`, `Jose San Juan`, `Juan De Los Santos`, `De La Cruz, Juan`, `Del Rosario, Maria Luisa`, `, De La Cruz`, `Cruz, Juan de la`, `JUAN DE LA CRUZ`, `juan de la cruz`, `Juan De Guzman Jr.`, `Del Rosario`, `La Cruz`, `San Juan`, `Juan Del`, `Maria La Paz Reyes`, `Dr. Jose De Leon`, `Del Pilar Santos`, `Juan Santos, De La` and `Ma. Teresa De Los Reyes`, each parsed in both trees under the three orders and `default`, `plain:fil`, `strict:fil` and `strict:fil+en` (`config_diff.build` names each), and the cells compared; the review's own grid of 39 names agreed in direction, and `config_diff.py diff default plain:fil --local all --match '(?i)\b(de|del|la|san|los)\b' --sample 4` read 0 of 61,947 names differently in either tree.
- **2026-11-03 #1005, strict-gap batch C, ca (Derek) — ca.toml TAKES THE CATALAN PARTICLES `de`, `del` AND `la`; NO DEFAULT OR PLAIN-CHOICE PARSE MOVES; UNDER STRICT `ca`, `del` AND `la` READ AS NEVER-GIVEN.** One of the five strict-gap batches decided in chat on 2026-11-03 ("All five, here in order"), on the re-ranked report (`tools/differential/config_diff.py gaps` on master f48c88a3; the batches' own entry is the 2026-11-03 #1005, the strict-gap batches entry). ca.toml held the conjunction `i` alone, so strict `ca` read Catalan names' particles (`de la Torre`, `del Castell`) as name words: over the records of Spain and Andorra it broke 626 records on `de`, 208 on `del` and 197 on `la`, and the language's 1,032 broken records fall to 198 on this change alone (188 with batch A's `jr`). It also loses fixes, 182 to 26: 160 records (PatentsView ES 71, ORCID ES 61, PSC Spanish 28), nearly all Spanish names whose filer boxed a devotional or particle element with the given name (`María del Carmen García`, given box 'María del Carmen'), which strict `ca` now reads as plain `ca` and the default do, having agreed with the boxes only by lacking the particles; the review found no Catalan name read worse than by the default (2026-11-03). Each entry copies es.toml's kind and properties (`de` states `may_be_given = false`).
- **What moves.** The compiled default does not (`tools/vocab/build.py` reports `_compiled.py` current). Over a grid of 16 Catalan names (each comma shape, cased, upper and lower spellings, `i` joining two surnames, a title and a credential beside them) under the three orders, the default, plain `ca` and strict `ca` + `es` move no cell; strict `ca` moves 35, toward the default's reading save the class batch B met: `del` and `la` hold a `given` reading in es.toml alone, so under strict `ca` they read as never-given particles (`Del Castell` → family 'Del Castell', where the default reads given 'Del' and reports particle-or-given), and `_reading_index.strict_ungated`'s reviewed table gains `ca: del, la`. The same class after a family comma loses the given name (`Castell, Del` → family 'Castell Del', `Torre, La` → family 'Torre La', where the default reads given 'Del' and 'La'), as the default already reads `la Torre, De`. A plain choice reads `Reading.languages` of `de` and `la` with `ca` among the files (the default too; plain `ca` names `ca` alone); under plain `ca` es.toml's `given` readings of `del` and `la` survive the preference, so `Del Shannon` still reports particle-or-given. Not added: the Catalan plural `dels` (`Pere dels Arcs` reads middle 'dels' under every choice, the default included; 2 names in the local corpora), no file holding it, so the report could not rank it.
- **Recompute.** `tools/differential/config_diff.py gaps ca` on the tree before and after; the grid is the 16 names `Jordi de la Torre`, `Joan del Castell`, `Montserrat Puig i Vila`, `de la Torre, Jordi`, `Del Castell, Joan`, `Puig, Joan de`, `, de la Torre`, `JORDI DE LA TORRE`, `jordi de la torre`, `Del Castell`, `La Torre`, `Maria de Lourdes Puig`, `Joan Miró i Ferrà`, `Dr. Jordi de la Torre`, `Anna de Puig, PhD` and `Josep Maria de Sagarra`, each parsed in both trees under the three orders and `default`, `plain:ca`, `strict:ca` and `strict:ca+es` (`config_diff.build` names each), and the cells compared; the review's own grid of 3,636 cells agreed (default, plain `ca` and strict `ca` + `es` unmoved), and `config_diff.py diff default plain:ca --local all --match '(?i)\b(de|del|la)\b' --sample 4 --readings` wrote byte-identical readings in both trees over 61,582 names.
2 changes: 1 addition & 1 deletion docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Release Log
- **Add a switch for names with a compound given name.** ``Parser(policy=Policy(two_given_names=True)).parse("Juan Carlos Pérez")`` gives given ``Juan Carlos``, where 1.4.0 through 2.8.0 gave given ``Juan``, middle ``Carlos``. Off by default: under it a middle name beside the given name joins it (``John Quincy Adams`` gives given ``John Quincy``). With ``two_family_names`` on too, the family takes its word first, so ``María José García López`` reads given ``María José``, family ``García López``. A ``Policy`` pickled by 2.8 loads with it off. :issue:`704`
- **Add Lexicon.given_names, given names two_family_names leaves out of the family.** ``Parser(policy=Policy(two_family_names=True), lexicon=Lexicon.default() | Lexicon(given_names={"luis"})).parse("José Luis Pérez")`` gives middle ``Luis``, family ``Pérez``, where the switch alone, in 2.8.0 as now, gives family ``Luis Pérez``. Empty by default, so the switch alone reads as before; an entry without accents matches the word with them. :issue:`959`
- **Add Token.reading, the reading the parse took for each word.** ``parse("John Smith PhD Jones").tokens[-1].reading`` is ``Reading('credential' as 'suffix', 'position')``, the reading 2.7's ``Ambiguity.alternative`` gave only for the other side of a fork, and ``ReadingKind`` gains ``patronymic``. The field is not compared, and a ``ParsedName`` pickled by 2.x loads with no readings. :issue:`754`
- **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('es', 'fil', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753`
- **Add Reading.languages, the language files that list a word the way it was read.** ``parse("Dr. Juan de la Vega III").tokens[2].reading.languages`` is ``('ca', 'es', 'fil', 'fr', 'it', 'nl', 'pt')`` for the particle ``de``, and ``Ambiguity.alternative`` names its files too; a reading by shape or position names none. Under a choice of languages it names the files the choice reads: ``parser_for(locales.PT)`` names ``('pt',)`` for that ``de``. It is not compared, and a ``Reading`` pickled by 2.7 or 2.8 loads with none. The first parse in a process that meets a listed word loads the reading index once: about 1 ms, or about 16 ms where Python writes no bytecode cache. :issue:`754`, :issue:`753`
- **Add Czech, Slovak, Slovenian and Croatian packs, locales.CS, SK, SL and HR.** ``parser_for(locales.CS).parse("Jan Novák, MUDr")`` gives title ``MUDr``, given ``Jan``, family ``Novák``, where 1.4.0 through 2.8.0 gave given ``MUDr``, family ``Jan Novák``. Each pack chooses its one language and turns on words the default does not read: ``ml.`` and ``st.`` after the name in all four (``Jiří Dienstbier ml.``), the Czech and Slovak academic titles and ``DrSc.``, and Czech ``Pan`` and ``roz.``. Choosing the language drops ``st``'s Saint readings (``St Clair Smith`` reads given ``St``), which choosing ``fr`` as well keeps. ``CS`` and ``SK`` also read two family names, so ``Alena Pešová Řehořová`` gives family ``Pešová Řehořová`` and two given names read as a given name and a family name (``Petr Pavel Novák`` gives family ``Pavel Novák``); a family comma keeps both. :issue:`753`
- **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753`
- **Add parser_for(..., strict=True), to read only the chosen languages' words.** ``parser_for(Locale(code="ko", lexicon=Lexicon.empty(), languages=frozenset({"ko"})), strict=True).parse("Kim Do Hyun")`` gives middle ``Do``, family ``Hyun``, where the default, as in 1.4.0 through 2.8.0, and a plain ``ko`` choice give family ``Do Hyun``, ``Do`` read as the Portuguese particle. Only the chosen languages' vocabulary files are read, with the words every language writes alike (``Dr``, ``PhD``, ``Mr``, the numerals) and the words written in other scripts; every other language's words leave, the rest of the English credentials and salutations included, so ``Kim Min Jun, MD`` gives given ``MD`` unless ``en`` is chosen too. A strict call that chooses no language raises ``ValueError``. ``python -m nameparser`` gains ``--strict``, and ``--locale`` takes a language tag no pack has (``--locale ko``). See :doc:`locales`. :issue:`753`, :issue:`1005`
Expand Down
4 changes: 2 additions & 2 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -1270,8 +1270,8 @@ file does hold turns that file's reading on, and names it:
... print(f"{tok.text}: {tok.reading.languages}")
Dr.: ('mul',)
Juan: ()
de: ('es', 'fil', 'fr', 'it', 'nl', 'pt')
la: ('es', 'fil', 'fr', 'it')
de: ('ca', 'es', 'fil', 'fr', 'it', 'nl', 'pt')
la: ('ca', 'es', 'fil', 'fr', 'it')
Vega: ()
III: ('mul',)

Expand Down
2 changes: 1 addition & 1 deletion nameparser/_types.py
Original file line number Diff line number Diff line change
Expand Up @@ -958,7 +958,7 @@ class Reading:
source: ReadingSource
#: The BCP 47 tags of the shipped vocabulary files that hold this
#: reading of the word, on in the parse's own ``Lexicon``
#: (``("es", "fil", "fr", "it", "nl", "pt")`` for the particle ``de`` by
#: (``("ca", "es", "fil", "fr", "it", "nl", "pt")`` for the particle ``de`` by
#: default, ``("pt",)`` under a ``pt`` choice, which drops the
#: others' readings of it; ``("mul",)`` for the title ``Dr.``), so
#: a choice of languages names the files it turns on and not those
Expand Down
6 changes: 3 additions & 3 deletions nameparser/config/_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -573,14 +573,14 @@
('dcvo', (('en', 'credential', ()),)),
('dd', (('en', 'credential', ()),)),
('dds', (('en', 'credential', ()),)),
('de', (('es', 'preposition', ()), ('fil', 'preposition', ()), ('fr', 'preposition', ()), ('it', 'preposition', ()), ('nl', 'article', ()), ('pt', 'preposition', ()))),
('de', (('ca', 'preposition', ()), ('es', 'preposition', ()), ('fil', 'preposition', ()), ('fr', 'preposition', ()), ('it', 'preposition', ()), ('nl', 'article', ()), ('pt', 'preposition', ()))),
("de'", (('it', 'preposition', ()),)),
('deacon', (('en', 'religious', ()), ('en', 'given', ()), ('en', 'family', ()))),
('dean', (('en', 'office', ()), ('en', 'given', ()), ('en', 'family', ()))),
('ded', (('en', 'credential', ()),)),
('degli', (('it', 'preposition', ()),)),
('dei', (('it', 'preposition', ()),)),
('del', (('es', 'preposition', ()), ('es', 'given', ()), ('fil', 'preposition', ()), ('it', 'preposition', ()))),
('del', (('ca', 'preposition', ()), ('es', 'preposition', ()), ('es', 'given', ()), ('fil', 'preposition', ()), ('it', 'preposition', ()))),
('dela', (('fil', 'preposition', ()), ('fil', 'given', ()))),
('delegate', (('en', 'office', ()),)),
("dell'", (('it', 'preposition', ()),)),
Expand Down Expand Up @@ -975,7 +975,7 @@
('kreivi', (('fi', 'nobility', ()),)),
('kt', (('en', 'credential', (('cased', 'Kt'),)),)),
("l'", (('fr', 'article', ()), ('it', 'article', ()))),
('la', (('es', 'article', ()), ('es', 'given', ()), ('fil', 'article', ()), ('fr', 'article', ()), ('it', 'article', ()))),
('la', (('ca', 'article', ()), ('es', 'article', ()), ('es', 'given', ()), ('fil', 'article', ()), ('fr', 'article', ()), ('it', 'article', ()))),
('lac', (('en', 'credential', ()), ('en', 'family', ()))),
('ladc', (('en', 'credential', ()),)),
('lady', (('en', 'nobility', (('attaches', ('justice',)),)),)),
Expand Down
12 changes: 12 additions & 0 deletions nameparser/vocab/ca.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,17 @@
schema = 1

[[word]]
text = "de"
preposition = { may_be_given = false, expands = "of: \"Jaume de Montcada\"", note = "strict ca broke 626 records on it while only es.toml and others held it (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 }

[[word]]
text = "del"
preposition = { expands = "of the: \"Joan del Castell\"", note = "strict ca broke 208 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 }

[[word]]
text = "la"
article = { expands = "the, fem. sg.: \"de la Torre\"", note = "strict ca broke 197 records (config_diff.py gaps on f48c88a3, decisions/vocabulary-model, 2026-11-03)", issue = 1005 }

[[word]]
text = "i"
[word.conjunction]
Expand Down
4 changes: 2 additions & 2 deletions tests/v2/pipeline/test_assemble.py
Original file line number Diff line number Diff line change
Expand Up @@ -482,8 +482,8 @@ def test_every_suffix_words_reading_alternative_and_index_kind_agree(
_EXPECTED_LANGUAGES = [
# a particle, by its particle rows only ('la' is a given name in es
# too, a reading the parse did not take)
("Dr. Juan de la Vega III", "de", ("es", "fil", "fr", "it", "nl", "pt")),
("Dr. Juan de la Vega III", "la", ("es", "fil", "fr", "it")),
("Dr. Juan de la Vega III", "de", ("ca", "es", "fil", "fr", "it", "nl", "pt")),
("Dr. Juan de la Vega III", "la", ("ca", "es", "fil", "fr", "it")),
("Dr. Juan de la Vega III", "Dr.", ("mul",)),
("Dr. Juan de la Vega III", "III", ("mul",)),
("John Smith PhD", "PhD", ("mul",)),
Expand Down
2 changes: 1 addition & 1 deletion tests/v2/test_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -2972,7 +2972,7 @@ def test_reading_languages_follow_the_choice() -> None:
assert _languages_of(pt, "Vincent van Gogh", "van") == ("nl",)
# with no choice, the files on by default, as before
assert _languages_of(Parser(), "Ana de Souza", "de") == (
"es", "fil", "fr", "it", "nl", "pt")
"ca", "es", "fil", "fr", "it", "nl", "pt")


def test_reading_languages_under_a_strict_choice_name_the_chosen_files() -> None:
Expand Down
1 change: 1 addition & 0 deletions tests/v2/test_reading_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,7 @@ def test_a_file_holding_no_reading_is_chosen_only_strictly() -> None:
#: vocabulary reading loses its gate: review each move.
STRICT_UNGATED = {
"ar": {"cheikh", "sultan"},
"ca": {"del", "la"},
"cs": {"st"},
"fil": {"del", "la", "san"},
"fr": {"la", "marquis", "marquise", "st"},
Expand Down
Loading