Skip to content

fix(Middleware): reject fields / page / filter an endpoint does not support (#261) - #313

Merged
thomasfosterau merged 1 commit into
trunkfrom
issue-261-fields-page-filter
Oct 9, 2026
Merged

thomasfosterau merged 1 commit into
trunkfrom
issue-261-fields-page-filter

Conversation

@thomasfosterau

Copy link
Copy Markdown
Owner

Summary

Closes #261

This implements the three owner decisions recorded on #261 on 2026-10-09. The include / sort part landed in #280, #283, #294, #300 and #306.

  • (a) 400 for an unsupported fields[…], page[…] or filter[…]. It works like include / sort since fix(Middleware): reject include / sort on an endpoint that does not support them (#261) #280: one error object per offending parameter with source.parameter naming it, the same check in Middleware.SchemaErrors, and the same precedence over 406 / 415. Any value is rejected, an empty one included, and so is the bare name (?page=2, ?filter=x, ?fields=title).
  • (b) allowedQueryParameters refuses the spec's family names. An entry include, fields, sort, page or filter throws InvalidAllowedQueryParameter when schemaErrorsLayer / layerWith builds the layer, and from checkQueryParameters.
  • (c) An empty vocabulary means unsupported, for these families too. That covers filter: true on a resource with no filterable field, filter: {}, page: {}, and a fields[TYPE] whose resource has no attribute.

fields is per resource type. A fields[TYPE] is supported only by its exact wire name. So a type the endpoint has no fieldset for, or one whose type has no attribute, is unsupported for that wire key only. The endpoint's other fields[TYPE] keep decoding (/posts?fields[posts]=title&fields[tags]= rejects only fields[tags]). A bare fields and a fields[TYPE][…] are rejected the same way. This follows the note on #261 from the #300 review: admitsNoValue took no type, and its "any member" struct rule would have rejected fields[articles] beside an empty fields[tags]. Mutation M7 below shows exactly that failure.

What the spec says

Quoted from the spec source, _format/1.1/index.md on json-api/json-api gh-pages (bde79ce):

  • L2124-2126, §Implementation-Specific Query Parameters (the general rule): "If a server encounters a query parameter that does not follow the naming conventions above, or the server does not know how to process it as a query parameter from this specification, it MUST return 400 Bad Request." This is the only 400 that applies to page and filter. Their sections only reserve the family (L1427-1428: "The page [query parameter family] is reserved for pagination. Servers and clients SHOULD use these parameters for pagination operations."; L1441-1442 says the same for filter).
  • L1323-1325, §Sparse Fieldsets: "If a client requests a restricted set of [fields] for a given resource type, an endpoint MUST NOT include additional [fields] in resource objects of that type in its response." So for fields, the old behaviour (ignore it and answer with every field) was itself non-compliant.
  • L1316-1317, the same section: "A client MAY request that an endpoint return only specific [fields] in the response on a per-type basis by including a fields[TYPE] query parameter." This is why support is decided per fields[TYPE].

Changes

  • src/internal/vocabulary.ts
    • FAMILIES (include, fields, sort, page, filter) is now the single runtime list. Family and isFamily derive from it, and Vocabulary's non-fields arm is Exclude<Family, "fields">.
    • admitsNoValue(query, name) now takes the wire name. For fields[TYPE], it reads the property that wire name decodes into: the one of that name, Query.schema's nested fields, or the property a Schema.encodeKeys renames to it. Within that property it counts only a vocabulary of that type, so a neighbouring type's fieldset never decides it. Other families behave as before.
    • vocabularyAnnotation is exported for Query.ts.
  • src/Middleware.ts
    • SPEC_FAMILIES / SUPPORT_FAMILIES / SUPPORT_FAMILY_NAMES are replaced by UNSUPPORTED: Record<Family, (parameter) => string>, an exhaustive per-family detail. The include / sort details are unchanged.
    • isSupported replaces the old known list. A fields name is admitted only by its exact declared name or a declared fields[ prefix. Any other name admits its whole family, as before.
    • supportedNames drops every declared name for which admitsNoValue holds.
    • The allow-list validation (refusal) refuses spec family names with their own message ("names a query parameter family JSON:API defines …").
    • JSDoc is updated on the module header, QueryParameterOptions.allowedQueryParameters, InvalidAllowedQueryParameter, QueryParameterCheck.declared, checkQueryParameters, SchemaErrors, schemaErrorsLayer, SchemaErrorsLive and layerWith.
  • src/Query.ts
    • Query.Fieldset publishes { family: "fields", type, values } on its literal schema. When the set is empty its rejection now reads This endpoint supports no sparse fieldset for type "tags". instead of Expected never, on a direct decode.
    • Query.schema's filter: true publishes the grammar's filterable field names on the decoded tree's schema.
    • Options.fields / page / filter docs mention the 400.
    • A non-empty fieldset's rejection is unchanged. It also gets no meta, because errorMeta's META_KEYS still lists only include / sort.
  • README.md: the "Reserved query parameters" allow-list paragraph and the "Query parameters" paragraph on spec families. No other section was touched; in particular, not the undeclared-body-members paragraph.

Behaviour check: example apps

I added a temporary test (not committed) to each example. It sends every GET endpoint, with ids 1 and 2, through 35–47 query strings: include / sort / fields[TYPE] for every resource type, bare fields, fields[nope], seven page[…] spellings, five filter spellings, unknown, fooBar. Each request went with and without a parameterised Accept. I compared status, headers (less date) and body between trunk's sources and this branch's.

Example Requests Identical Changed
blog 840 464 376
github 1824 800 1024
northwind 3948 1308 2640

Every changed response is an intended new 400. The script asserted that each one is a 400 whose errors is exactly one object, naming the probed parameter in source.parameter, with the "does not support the … parameter" detail. On trunk these requests answered 200, 404 (an id that doesn't exist: the check now runs before the handler) or 406 (the parameterised Accept, which this 400 now takes precedence over). Responses for include, sort, unknown and fooBar are byte-identical. These are the newly rejected parameters per endpoint, with the ones whose responses are unchanged:

blog

  • GET /articles: fields[articles], fields[comments], fields[people], fields[tags], bare fields, fields[nope]; still decoded: filter, page
  • GET /articles/1: filter[…], page[…], bare fields, fields[nope]; still decoded: fields[articles], fields[comments], fields[people], fields[tags]
  • GET /articles/1/author: fields[articles], fields[comments], fields[people], fields[tags], filter[…], page[…], bare fields, fields[nope]
  • GET /articles/1/comments: fields[articles], fields[comments], fields[people], fields[tags], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /articles/1/relationships/comments: fields[articles], fields[comments], fields[people], fields[tags], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /search: bare fields, fields[nope]; still decoded: fields[articles], fields[comments], fields[people], fields[tags], filter, page

github

  • GET /issues: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], bare fields, fields[nope]; still decoded: filter, page
  • GET /issues/1: fields[pulls], filter[…], page[…], bare fields, fields[nope]; still decoded: fields[issues], fields[labels], fields[repositories], fields[users]
  • GET /issues/1/comments: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /issues/1/relationships/labels: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], page[…], bare fields, fields[nope]
  • GET /pulls: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], bare fields, fields[nope]; still decoded: filter, page
  • GET /pulls/1: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], page[…], bare fields, fields[nope]
  • GET /repositories: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], bare fields, fields[nope]; still decoded: filter, page
  • GET /repositories/1: fields[issues], fields[labels], fields[pulls], filter[…], page[…], bare fields, fields[nope]; still decoded: fields[repositories], fields[users]
  • GET /repositories/1/owner: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], page[…], bare fields, fields[nope]
  • GET /search: fields[pulls], bare fields, fields[nope]; still decoded: fields[issues], fields[labels], fields[repositories], fields[users], filter, page
  • GET /users: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /users/1: fields[issues], fields[labels], fields[pulls], fields[repositories], fields[users], filter[…], page[…], bare fields, fields[nope]

northwind

  • GET /categories: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /categories/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /customers: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /customers/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /employees: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /employees/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /employees/1/relationships/territories: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /orders: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /orders/1: fields[categories], fields[products], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]; still decoded: fields[customers], fields[employees], fields[orders], fields[shippers]
  • GET /orders/1/lineItems: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /orders/1/relationships/shipper: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /products: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /products/1: fields[customers], fields[employees], fields[orders], fields[shippers], fields[territories], filter[…], page[…], bare fields, fields[nope]; still decoded: fields[categories], fields[products], fields[suppliers]
  • GET /products/1/supplier: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /search: fields[employees], fields[orders], fields[shippers], fields[territories], bare fields, fields[nope]; still decoded: fields[categories], fields[customers], fields[products], fields[suppliers], filter, page
  • GET /shippers: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], bare fields, fields[nope]; still decoded: page
  • GET /shippers/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /suppliers: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /suppliers/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
  • GET /territories: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], bare fields, fields[nope]; still decoded: filter, page
  • GET /territories/1: fields[categories], fields[customers], fields[employees], fields[orders], fields[products], fields[shippers], fields[suppliers], fields[territories], filter[…], page[…], bare fields, fields[nope]
    "Unchanged" means every probe of that family answered byte-identically, codec 400s included. One consequence to note: GET /orders/10248?include=employee.territories&fields[territories]=region now answers 400 on northwind. fields: true declares no fieldset for a type that only a depth-2 include path reaches. Filed as feat(Query): fields: true should declare a sparse fieldset for every type an include path reaches #311 (see Notes).

Tests

The tests were written first, over HTTP, in src/Middleware.query.test.ts. On unfixed trunk sources, 98 of 218 tests in that file failed: every new rejection case, the allow-list refusals, and the checkQueryParameters cases.

  • New block, "fields, page and filter an endpoint does not support (bug: ?include= / ?sort= on an endpoint that does not support them is ignored, though the spec makes it a 400 #261)". 22 rejection cases run under each of Middleware.layer, Middleware.layerWith({ allowedQueryParameters: ["lang"] }) and schemaErrorsLayer beside ContentNegotiationLive. Each asserts the exact error document and that no handler ran. The cases cover:
    • Undeclared: each family, empty values, bare names, a nested filter[…][…], all three at once.
    • Empty vocabulary: filter: true over no filterable field, filter: {}, page: {}, a fieldset type with no attribute.
    • Partly declared fields: a type with no fieldset, an empty-vocabulary type beside a supported one, a bare fields and a fields[TYPE][x] beside declared fieldsets, another type than the fetch declares.
  • Also in that block:
    • a DELETE with no query schema;
    • request-order interleaving with sort / unknown;
    • supported families still decoding, with the decoded query asserted;
    • an undeclared member of a declared page family still dropped;
    • layerHostNegotiated.
  • Precedence over 406 / 415, under Middleware.layer and schemaErrorsLayer beside ContentNegotiationLive: an undeclared page[limit] and an empty-vocabulary filter[name] win over the 406; an empty-vocabulary fields[tags] and an undeclared fields[posts] win over the 415. Each test also checks the 406 / 415 without the parameter.
  • Hand-built queries (SchemaErrorsLive):
    • a union of fieldsets, where one member's tags is non-empty;
    • a union with an empty fieldset plus a page member;
    • a Schema.suspend query;
    • a fieldset under its own wire key "fields[tags]";
    • an empty fieldset renamed by Schema.encodeKeys, and a non-empty one;
    • an open fields[ record;
    • a Schema.Record(Schema.String, …), for which the whole check stands aside: ?fields[x]=a&page[y]=b&filter[z]=c answers 200.
  • Allow-list: each of the five family names is refused under schemaErrorsLayer, layerWith and checkQueryParameters.
  • checkQueryParameters: the three families undeclared; fields[TYPE] admitted only by its own name or a fields[ prefix.
  • Replaced tests. The allow-listed-include / sort tests from docs, test: #280 review follow-ups (include / sort 400 citations, precedence tests) #283 / fix: treat an include / sort with an empty vocabulary as unsupported (#282) #294 are gone, because those layers now throw at build time. Their replacement is the refusal cases. They covered "admits a family listed in allowedQueryParameters, which the endpoint then ignores", "still decodes a family listed …" and "says the endpoint supports none when an allow-listed empty vocabulary rejects a value". That codec detail is now reachable only on a direct decode, which src/Query.test.ts still covers. The old "still ignores the undeclared ?fields[people]=…" case is reversed into the new block.
  • src/Query.test.ts: the new Query.Fieldset empty-set detail (direct, and through Query.schema), plus a non-empty control.

Mutation checks

Each row applied one mutation to the final code, ran vp test run src/Middleware.query.test.ts src/Query.test.ts src/Middleware.test.ts (482 tests), then reverted it.

# Mutation Failing
T0 Trunk sources with the new tests (Middleware.query.test.ts alone) 98 of 218
M1 Allow-list accepts spec family names 15 (every refusal case)
M2 fields matched per family, not per wire key 18 (type-without-fieldset, bare fields, fields[TYPE][x], the checkQueryParameters per-key case, …)
M3 A declared fields[ prefix not honoured 2 (the checkQueryParameters prefix case, the open fields[ record)
M4 fields / page / filter always known (trunk's SPEC_FAMILIES) 82
M5 Empty vocabularies count as declared 45
M6 Generic #148 detail for every family 117
M7 Fieldset type ignored (any fields vocabulary counts) 5 (fields[posts] beside an empty fields[tags] is rejected: the #300-review misclassification, ×3 layers, plus two supported cases)
M8 Property not looked up by the family name (Query.schema's nested fields) 12
M9 Property not looked up by the exact wire name 1 (an empty fieldset under its own wire key)
M10 Rename fallback keyed on fields, not fields[TYPE] 2 (renamed empty fieldset; fieldset under its own wire key)
M11 Query.Fieldset publishes no fields vocabulary 18
M12 The filter: true grammar publishes no vocabulary 8 (every filter: true over-no-field case, and the 406-precedence case that uses it)

Warn-tier lint. vp lint per changed file, trunk → head: Middleware.ts 36 → 36, Middleware.query.test.ts 1 → 1, Query.ts 144 → 144, Query.test.ts 63 → 63, internal/vocabulary.ts 0 → 0. The per-rule counts are identical too. I removed two new sites I had introduced (imperative-loops, avoid-native-object-helpers), one as cast and one T | undefined.

Declarations. dist/*.d.ts gains no bare import "effect/…" lines. None of the new internals (FAMILIES, isFamily, vocabularyAnnotation) appears in the emitted declarations. The Family / isFamily there are Resource's own.

Verification

All gates ran in a fresh clone, one at a time, in AGENTS.md order, on the tree committed as the head commit:

  • pnpm install --frozen-lockfile: pass
  • vp run build: pass ("Build complete")
  • vp check: pass ("Found 0 errors and 4332 warnings in 132 files": existing warn-tier debt; no per-file or per-rule count rose)
  • vp run check (sources, type-level tests, examples, emitted declarations): pass ("check: every type gate green")
  • vp run -r test:unit: pass. Root: 2817 tests. Examples: 45 / 41 / 54. The conformance kit still passes against all three, with unchanged KNOWN_FAILURES.
  • vp run docgen (every JSDoc @example compiled and executed): pass ("Docs generation succeeded!", "every class example compiled and ran")
  • fallow audit --base origin/trunk: pass ("No issues in 10 changed files"; the gate excluded 3 inherited findings: the duplicate Sort export, Query.schema cyclomatic 18 and Middleware.sourceOf CRAP 43.1, none introduced here). Agent gate fallow audit --format json --quiet --explain --gate-marker agent: verdict pass (0 introduced).

pnpm audit --audit-level high (CI only) was not run locally; this changes no dependency.

Changeset

  • .changeset/reject-unsupported-fields-page-filter.md, patch. It opens with a bold Behaviour change line saying that a request that used to succeed can now answer 400. It covers fields per type, empty vocabularies, the allow-list refusal, checkQueryParameters, the precedence over 406 / 415 / 404, and the new Query.Fieldset empty-set detail.
  • Three pending changesets would now contradict this, so I amended them:
    • reject-unsupported-include-sort.md said an allow-listed include / sort is admitted and that fields / page / filter are still ignored;
    • reject-reserved-query-parameters.md said fields / page / filter are left to the schema, and its allow-list sentence didn't mention spec names;
    • empty-vocabulary-unsupported.md said an allow-list entry reaches the codec detail.

Notes

Owner questions (defaults implemented, flag if you want otherwise):

  1. An undeclared member of a family the endpoint does declare is still dropped by the query schema, as include[x] beside a declared include is. Examples: page[size] beside Query.Page.Offset, or filter[x] beside a filter: { q } map. The decision on bug: ?include= / ?sort= on an endpoint that does not support them is ignored, though the spec makes it a 400 #261 covers "an endpoint that doesn't support that family", so I kept these family-level and pinned it with a test. Rejecting them under the general rule would be a further behaviour change. fields is the exception, because its family is per type.
  2. An empty fields[TYPE]= for a type with no attribute is now a 400. The spec gives an empty value a meaning ("An empty value indicates that no fields should be returned", L1321). I applied decision (c) and the coordinator's guidance as written: an empty vocabulary is unsupported, empty value included, as for ?include=.
  3. The rename fallback errs toward "supported", as fix: read include / sort support off the wire name (#300 follow-up) #306's does. Take a hand-built struct that renames two fieldsets with Schema.encodeKeys, one of them empty. The AST doesn't say which property is which wire key, so the empty one's fields[TYPE] is left to its codec (a 400 from the codec rather than this one). A single renamed fieldset is decided exactly (tested).

Follow-ups filed:

#261's done-when:

#310 extends item 3 to the families decided later. I read it as a follow-up rather than as part of #261's original done-when, hence Closes #261. If you count it as part of item 3, change this to Part of #261.

Not touched: src/internal/schemaExt.ts, .changeset/reject-undeclared-*, the README undeclared-body-members paragraph (#304 follow-up), and the conformance kit (#168).

🤖 Generated with Claude Code


Generated by Claude Code

… not support (#261)

Owner decisions on #261 (2026-10-09):

- An undeclared `fields[TYPE]`, `page[…]` or `filter[…]` is a 400 naming
  the parameter, with the same precedence over 406/415 as `include` /
  `sort` (#280). `fields` is per resource type: a `fields[TYPE]` for a
  type the endpoint has no fieldset for, or whose type has no attribute,
  is unsupported for that wire key only.
- `allowedQueryParameters` refuses `include`, `fields`, `sort`, `page`
  and `filter` with `InvalidAllowedQueryParameter`.
- An empty vocabulary means unsupported: `filter: true` over no
  filterable field, `page: {}`, `filter: {}`, a fieldset type with no
  attribute.

`Family` (now a runtime `FAMILIES` list in `internal/vocabulary.ts`) is
the single source of truth, replacing `SPEC_FAMILIES` /
`SUPPORT_FAMILIES`. `admitsNoValue` takes the wire name and reads a
`fields[TYPE]` off that type's fieldset alone. `Query.Fieldset` and the
`filter: true` grammar publish their vocabularies.

Closes #261

Claude-Session: https://claude.ai/code/session_0187g2QpzaHqEzX8GJxiJBVc
Co-Authored-By: Claude <noreply@anthropic.com>

Copy link
Copy Markdown
Owner Author

Owner decision on O1 (2026-10-09): the strict 400 for a fields[TYPE] naming a type the endpoint can never return stays. #311 (fieldsets for every type an include path reaches) lands before 0.17.0. Recorded on #311.


Generated by Claude Code

Copy link
Copy Markdown
Owner Author

Owner decision on O2 (2026-10-09): reject with 400. An undeclared member of a declared filter map or of the declared page keys (filter[stauts], page[sise]) gets the same 400 as an undeclared family. Filed as #321; it is queued behind #317, since both touch src/Middleware.ts.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: ?include= / ?sort= on an endpoint that does not support them is ignored, though the spec makes it a 400

2 participants