Skip to content

engine(content-lane): add the missing documentation_url snake_case alias and make the urlFields pairing assertion derived #9992

Description

@JSONbored

⚠️ Definition of Done: this issue must be completed in full, in a single PR. Do not split this
work across multiple PRs, and do not defer any Deliverable below to a follow-up issue. A PR that
satisfies only some of the Deliverables, stubs a required test, or leaves a checkbox
partially-done does NOT resolve this issue and will be closed.

Context

AWESOME_CLAUDE_CONTENT_SPEC declares that every camelCase URL-bearing frontmatter key also has a
snake_case alias, because the curated list legitimately accepts both conventions. The comment on
sourceUrlFields states the invariant explicitly.

packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:94:

  urlFields: new Set([
    "documentationUrl",
    "docsUrl",
    "downloadUrl",
    "githubUrl",
    "packageUrl",
    "repoUrl",
    "repositoryUrl",
    "sourceUrl",
    "websiteUrl",
    "docs_url",
    "download_url",
    "github_url",
    "package_url",
    "repo_url",
    "repository_url",
    "source_url",
    "website_url",
  ]),

Nine camelCase keys, eight snake_case aliases. documentation_url is missing.

The same gap is in sourceUrlFields at packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:123,
whose own comment says:

    // snake_case aliases, matching urlFields one-for-one (#7250): source-evidence.ts read only the camelCase
    // names, so an entry using a legitimately-listed snake_case key (e.g. the canonical `source_url`) was visible
    // to duplicates.ts but invisible to the source-evidence gate.

"one-for-one" is not true: documentationUrl has no partner in either set.

Two live consumers read these sets by literal parsed key, so a submitted entry that spells the field
documentation_url is invisible to both:

  • src/review/content-lane/duplicates.ts:288.filter(([key]) => spec.urlFields.has(key)). The entry's
    documentation URL never enters ContentDuplicateSignals.urls, so a duplicate submission whose only
    distinguishing URL is documentation_url is not compared against the corpus.
  • src/review/content-lane/source-evidence.ts:273for (const field of spec.sourceUrlFields). The
    documentation URL is never extracted, so it is never fetch-verified as source evidence.

The regression test that is supposed to pin this invariant cannot catch it, because it enumerates the
aliases by hand instead of deriving them. test/unit/content-lane-source-evidence.test.ts:65:

    // Every snake_case alias urlFields carries is now recognized here too.
    for (const field of ["docs_url", "download_url", "github_url", "package_url", "repo_url", "repository_url", "source_url", "website_url"]) {
      expect(AWESOME_CLAUDE_CONTENT_SPEC.sourceUrlFields).toContain(field);
      expect(AWESOME_CLAUDE_CONTENT_SPEC.urlFields.has(field)).toBe(true);
    }

The sibling assertion for protectedFrontmatterFields at test/unit/content-lane-duplicates.test.ts:157
already does this correctly — it walks the whole set and derives the snake_case name — which is why the
protected-fields set has no such gap.

documentation_url appears nowhere in the repository.

Requirements

  • AWESOME_CLAUDE_CONTENT_SPEC.urlFields must contain "documentation_url".
  • AWESOME_CLAUDE_CONTENT_SPEC.sourceUrlFields must contain "documentation_url", placed in the
    snake_case alias block alongside the existing eight, preserving the file's existing ordering
    convention (camelCase block first, then the snake_case block).
  • The enumerated alias loop at test/unit/content-lane-source-evidence.test.ts:65 must be replaced by a
    DERIVED assertion over the whole set — for every member of urlFields that is camelCase (contains an
    uppercase letter), its field.replace(/[A-Z]/g, (l) => "_" + l.toLowerCase()) form must also be present in
    urlFields AND in sourceUrlFields. A hard-coded list with documentation_url appended does NOT satisfy
    this bullet: the point is that the next added URL field cannot reopen the same gap.
  • sourceUrlFields ordering must stay meaningful: its own doc comment says it is read "in extraction ORDER"
    by the source-evidence gate, so the new alias is appended to the snake_case block, not interleaved into the
    camelCase block.
  • What must NOT change: distributionSourceFields, primaryCanonicalSourceFields,
    protectedFrontmatterFields, domainOnlyExclusions, multiEntryCatalogUrls, and
    sourceUrlListFields are all correct as shipped and must be left byte-identical.
    documentationUrl must remain a scalar-only field (it must NOT be added to sourceUrlListFields) —
    test/unit/content-lane-source-evidence.test.ts:93 depends on that.

⚠️ Required pattern: mirror the derived pairing assertion at
test/unit/content-lane-duplicates.test.ts:157 (toSnakeCase walked over the entire set), which is
already the working precedent in this repo for exactly this invariant. What does NOT satisfy this issue:
(a) adding "documentation_url" to the two sets and leaving the hand-enumerated test loop in place;
(b) introducing a generic runtime "normalize any key to camelCase before lookup" layer in
duplicates.ts / source-evidence.ts — that changes matching behaviour for every field in every spec and
is far wider than this defect; (c) a test-only PR that asserts the gap exists without closing it.

Deliverables

  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts"documentation_url" added
    to both urlFields and sourceUrlFields.
  • test/unit/content-lane-source-evidence.test.ts — the enumerated alias loop replaced with a derived
    loop over AWESOME_CLAUDE_CONTENT_SPEC.urlFields, asserting for every camelCase member that its
    snake_case form is in urlFields and in sourceUrlFields. This test must fail against the current
    content-repo-spec.ts and pass after the fix.
  • test/unit/content-lane-source-evidence.test.ts — a regression test named for this bug asserting
    extractSubmittedSourceUrls on frontmatter containing only
    documentation_url: https://docs.acme.example/guide returns exactly one entry
    { field: "documentation_url", url: "https://docs.acme.example/guide" }.
  • test/unit/content-lane-duplicates.test.ts — a regression test asserting
    extractContentDuplicateSignals on an entry whose only URL frontmatter key is documentation_url
    produces a non-empty urls array containing the normalized URL (today it is []).
  • A new engine-package test at packages/loopover-engine/test/content-repo-spec.test.ts (this file does
    not exist yet; create it, importing AWESOME_CLAUDE_CONTENT_SPEC from
    ../dist/review/content-lane/content-repo-spec.js per the convention in
    packages/loopover-engine/test/content-lane-flag.test.ts) asserting the SAME derived camelCase ↔
    snake_case pairing invariant over urlFields and sourceUrlFields.

All Deliverables above are required in a single PR. A PR that satisfies only some of them — for example
one that adds "documentation_url" to the two sets and adds the root test/unit assertions but skips the
new packages/loopover-engine/test/content-repo-spec.test.ts file — does not resolve this issue.

Test Coverage Requirements

This repo enforces 99%+ Codecov patch coverage, branch-counted. vitest.config.ts's
coverage.include covers src/**/*.ts and packages/loopover-engine/src/**/*.ts — the touched path
packages/loopover-engine/src/review/content-lane/content-repo-spec.ts IS measured. The change adds set
members rather than branches, but the two consumers it re-enables (duplicates.ts:288's
spec.urlFields.has(key) filter and source-evidence.ts:273's field loop) have both arms exercised only
if the new tests assert BOTH a documentation_url entry being picked up AND an unrelated non-URL key still
being excluded; assert both.

Engine lines are credited by two uploads whose hits are unioned — add the test to
packages/loopover-engine/test/** as well as any root test/** coverage, or the patch gate can still fail.
That is precisely why the new packages/loopover-engine/test/content-repo-spec.test.ts file is a required
Deliverable and not optional.

Expected Outcome

A curated-list entry that spells its documentation link documentation_url — a spelling the spec already
accepts for eight sibling URL fields — is seen by duplicate detection and fetch-verified by the
source-evidence gate, exactly like documentationUrl. The pairing invariant is enforced by a derived
assertion in both the root and engine test suites, so the next URL field added to the spec cannot silently
reintroduce a half-registered alias.

Links & Resources

  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:94urlFields, missing documentation_url
  • packages/loopover-engine/src/review/content-lane/content-repo-spec.ts:123sourceUrlFields and its "matching urlFields one-for-one" comment
  • src/review/content-lane/duplicates.ts:288urlFields consumer
  • src/review/content-lane/source-evidence.ts:273sourceUrlFields consumer
  • test/unit/content-lane-source-evidence.test.ts:65 — the hand-enumerated alias assertion that cannot catch this
  • test/unit/content-lane-duplicates.test.ts:157 — the derived pairing assertion to mirror

Metadata

Metadata

Assignees

No one assigned

    Labels

    gittensor:bugGittensor-scored bug fix — scores a 0.05x multiplier.help wantedExtra attention is needed

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions