Japanese is the one CJK case that can't ship as bundled constants (#271): JMnedict contains ~138,500 surname entries, both surnames and given names run 1–3+ kanji, and the hard cases are ones where multiple splits are dictionary-valid (2+2 vs 3+1 vs 1+3) — membership lists can't resolve them; frequency statistics are required.
Proposal: rather than bundling data, make the unspaced-name segmenter (#271) pluggable and delegate Japanese to an optional dependency:
pip install nameparser[ja] # pulls namedivider-python
with a lazy import raising a helpful "install nameparser[ja]" error if the feature is enabled without the extra. namedivider-python is MIT-licensed, actively maintained, purpose-built for this exact task, and reports 99.91% accuracy.
Rejected alternative — JMnedict as the data source: the pip-native packaging (jamdict + jamdict-data) was last compiled in April 2021 and bundles far more than surnames; fetching EDRDG XML at install time breaks offline installs; JMnedict has no frequency data so it can't beat namedivider on the ambiguous splits; and its CC BY-SA license raises bundling questions our LGPL core never has to answer if the data stays in a third-party package.
A pluggable hook also means users can swap in better segmenters without waiting on us. Depends on the segmentation hook from #271; core nameparser stays zero-dependency. Related: #34, #270.
Japanese is the one CJK case that can't ship as bundled constants (#271): JMnedict contains ~138,500 surname entries, both surnames and given names run 1–3+ kanji, and the hard cases are ones where multiple splits are dictionary-valid (2+2 vs 3+1 vs 1+3) — membership lists can't resolve them; frequency statistics are required.
Proposal: rather than bundling data, make the unspaced-name segmenter (#271) pluggable and delegate Japanese to an optional dependency:
with a lazy import raising a helpful "install nameparser[ja]" error if the feature is enabled without the extra. namedivider-python is MIT-licensed, actively maintained, purpose-built for this exact task, and reports 99.91% accuracy.
Rejected alternative — JMnedict as the data source: the pip-native packaging (
jamdict+jamdict-data) was last compiled in April 2021 and bundles far more than surnames; fetching EDRDG XML at install time breaks offline installs; JMnedict has no frequency data so it can't beat namedivider on the ambiguous splits; and its CC BY-SA license raises bundling questions our LGPL core never has to answer if the data stays in a third-party package.A pluggable hook also means users can swap in better segmenters without waiting on us. Depends on the segmentation hook from #271; core
nameparserstays zero-dependency. Related: #34, #270.