Skip to content

Add the govuk-people local corpus: GOV.UK people's display names - #962

Merged
derek73 merged 2 commits into
masterfrom
feat/govuk-people-corpus
Oct 31, 2026
Merged

derek73 merged 2 commits into
masterfrom
feat/govuk-people-corpus

Conversation

@derek73

@derek73 derek73 commented Oct 31, 2026 •

Copy link
Copy Markdown
Owner

A new unscored local corpus, govuk-people (tools/differential/local_govuk.py), the third source the 2026-10-30 titled-name survey ranked. It holds the distinct display names of every person GOV.UK has a page for (ministers, senior officials, ambassadors, military chiefs, past prime ministers), paged from the search API filtered to the person format. Each name is the department's own display form, written with the British title run in front and post-nominals behind.

What it holds (fetched 2026-10-30, data date 2026-10-31T01:49:34Z)

  • 6,021 distinct names, held by 6,125 people. 99 names are held by two people or more.
  • parse() read a title in 1,293 (21.5%), a suffix in 1,261 (20.9%), and one or both in 2,093 (34.8%).
  • A raw count by vocabulary roughly agrees: 1,290 open with a listed title word (or The), and 1,149 end in a listed post-nominal.
  • Leading runs, by first word: The Rt Hon 272, Dr 229, Professor 227, Sir 147, Lord 94, Baroness 63, Dame 48, Mr 39, then military ranks.
  • 83 names hold a comma.

The fetch

  • It asks for order=title. Paged without an order, the search returned one person twice and skipped another across its 6,125 results.
  • It refuses:
    • a page stating a different total than the first;
    • a person paged twice (keyed by the page path, which is read and then dropped);
    • a set paged short of the total, including an empty page before the total;
    • results of another document type, or with no title, past 1 in 1,000.
  • The search states no date, so the data date is the fetch's own start time, in UTC.
  • A fetch took 1.7s. A second fetch the same day gave the same corpus.

Licence

  • GOV.UK terms (last updated 3 September 2024, read 2026-10-30): content is "published under the Open Government Licence".
  • OGL v3: "This licence does not cover: personal data in the Information".
  • So the names stay in the cache and are never committed, the same caution as UK MNIS in the 2026-10-27 survey. The tests use invented names only.

Changes

  • local_govuk.py, plus a SOURCES row in local_corpora.py.
  • tests/v2/test_local_govuk.py: 17 offline tests with a stubbed network. Each refusal has a test. The skip share is tested at its shipped value, where 1 skipped result in 1,000 passes and 2 are refused. Each of seven guard mutations (paged-twice, count, total-moved, empty-page, share, type-skip, order) fails at least one test.
  • test_no_module_imports_a_sibling_constant now lists the new module.
  • tools/differential/README.md "Local corpora": a row in the table, a paragraph, and the module in the module list.
  • AGENTS.md "Validating a new parsing rule": one sentence saying a change to English titles, honorifics or post-nominals reads this corpus.

A sibling Czech corpus and a Mexico DOF corpus register in the same lists, so whichever merges second rebases onto a one-line conflict.

Systematic misreadings it shows (counts only; no parser change here)

  • KC (King's Counsel) is in no vocabulary. It reads as the family name in 36 of the 42 names that hold it, for example Jane Doe KC → middle Doe, family KC.
  • Other unlisted British post-nominals end the family in 27 more names: FREng, FIET, DL, KFSM, LLB, FRS, FRSA, PC, HMI and others. After a comma, the same words reverse the name in 5 names, for example John Smith, DL → given DL, family John Smith.
  • Air Vice-Marshal, hyphenated, is not a title. The spaced form is. All 6 hyphenated names miss it.
  • Territorial peerage designations with a two-word place split, for example Baroness Doe of Kings Heath → given Doe of Kings, family Heath. This hits 9 names, while the one-word form reads whole.
  • A peerage written after the name and a comma reverses it, for example John Smith, 3rd Earl of Ashby → family John Smith, title 3rd. This hits 16 historical prime ministers.

Issue drafts are held for approval and not filed.

Checks

  • uv run pytest: 33,532 passed.
  • uv run mypy and uv run ruff check: clean.
  • test_local_govuk.py also passes on Python 3.15.
  • One real fetch, plus snapshot.py --local govuk-people --record: 6,021 readings recorded, and a check matches.

🤖 Generated with Claude Code

A new unscored local corpus, tools/differential/local_govuk.py: the
distinct display names of every person GOV.UK has a page for, paged from
the site's search API filtered to the person format. The department's
own display form, as written, with the British title run in front and
post-nominals behind: of 6,021 names (6,125 people) on 2026-10-30,
parse() read a title in 1,293, a suffix in 1,261, one or both in 2,093.

The fetch orders by title, since unordered paging returned one person
twice and skipped another, and refuses a total that moves between
pages, a person paged twice, a set paged short of the total, and
results of another type or with no title past 1 in 1,000. The OGL v3
does not cover personal data, so the names stay in the cache.

Registered in local_corpora.SOURCES; README "Local corpora" and
AGENTS.md's "Validating a new parsing rule" say when to read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ay name

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit b5f5656 into master Oct 31, 2026
13 checks passed
derek73 added a commit that referenced this pull request Oct 31, 2026
…ide with (#965) (#976)

* Add British post-nominals to the vocabulary, KC gated by its Nepali surname (#965)

GOV.UK's display names (govuk-people, #962) end in post-nominals no
vocabulary listed, read as the family name (`Jane Doe KC` -> family
'KC') or reversing a comma name (`Jane Doe, CChem` -> given 'CChem').

- `kc` is a credential written as a word (acronym = false) with name
  readings, gated as a generation word is (rules/S11): `John Smith KC`
  reads the suffix, `Ram KC`, `Ram Bahadur Kc` and `K.C.` keep the
  Nepali surname. `credential` joins GATED_WORDS, classify tags a gated
  credential with its kind and suffix_kind reads it first, so KC fills
  the credentials view, not generation.
- 18 credentials added (FRS, LLB, CChem, FRAeS, MRPharmS, ...), with
  case masks where mixed case; FIET and FREng gated by their surnames
  (Fiet, Freng); MoID refused, Moid being a given name.
- rules S11, S6 and S3 amended; decisions/S11 and
  decisions/suffix-acronym-collisions record the choice, measurements
  and accepted costs; decisions/S8 holds #965's join-order question
  as Open.

PC and DL are left for a separate decision.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Pin a gated credential's reading and alternative (#965)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Add PC and DL as gated credential words; fix the design-docs review (#965)

- `pc` and `dl` are credentials written as words beside family
  readings, gated as `kc` is: `Jane Doe, DL` and `Jane Doe PC` read
  the suffix, `Ravi PC` and `Anna DL` keep the family, the dotted
  spellings read as master did.
- rules/R8 says a shipped credential reads as one even where a caller
  lists it in suffix_words_ambiguous; S11's Then, S6 and the sibling
  bullets (C4, C12, S2, S5, S7) name the class as a suffix word that
  is also a name, not a generation word.
- decisions: KC's govuk count is 27, not 31; S8's Open example uses an
  unlisted word; FREng's mask limit is an accepted cost and an Open
  line; PC/DL's costs, including the Romanian `Dl` force view.
- examples drawn from real corpus records replaced (Gita Bahadur KC);
  check 11's narrower refusals each keep a mutation row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the second design-docs review of #965

- rules/R8: a shipped credential listed in suffix_words_ambiguous reads
  as a credential, and listed in agnomens as an agnomen (not 'either
  set'); decisions/R8 points at decisions/S11's entry.
- decisions/S11: the masks' default capitalized() moves, per corpus,
  for kc and for pc/dl, and the hyphenated 'Pc-' force view.
- decisions/suffix-acronym-collisions: tse-donors moves one case view.
- en.toml: the kc and pc notes point at decisions/S11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the review of #976: freng's mask, MoID's reason, counts (#965)

- freng: drop the `FREng` mask, which respelled the surname ('ANNA
  FRENG' -> 'Anna FREng'), and refuse it with `excluded = { cased }`
  as meng does; a test holds FRENG and FIET to their title case.
- moid: the refusal stands, its reason rewritten on the measurement
  (gated or refused, no name moves in fec, govuk, orcid or psc; gated
  it would gain only MOID and the comma, and cost SYED ABDUL MOID).
- decisions/S11: qc is a credential acronym, not a plain word; the
  mask's case views recounted per corpus with the recipe stated;
  `John Smith AO KC` moved out of the gains, to S8's open shape.
- decisions/suffix-acronym-collisions: orcid gains 3, FREng 2 kept and
  1 newly exposed, ukec's dl moves; FREng's Open points at
  decisions/S6, which now holds it; vocabulary-exclusions names the
  gated-word mask exception.
- rules/S11: When covers a word written as initials; credentials rows
  pin the gated credential's view.
- en.toml: ORCID box counts on acma, llb, mms, cchem and hmi.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Fix the fix-commit review of #976: freng's spelling, C-i's initials (#965)

- decisions/suffix-acronym-collisions: freng carries no mask; the S6
  reading applies to GOV.UK's mixed-case spelling FREng, said so in the
  Standing lines and the entry.
- decisions/vocabulary-collisions: a dated entry and Standing line read
  C-i's ground to include a person's initials written where the claim
  acts, with the same per-word count a name needs (pc, dl); rules/S11's
  Because says so.
- decisions/S11: the whole-word predicate behind the case-view counts
  is stated beside them, with DL's orcid figure under the other count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 deleted the feat/govuk-people-corpus branch November 4, 2026 20:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant