Skip to content

feat(Conformance): skip and requestTimeout options (#302) - #366

Merged
thomasfosterau merged 2 commits into
trunkfrom
issue-302-skip-and-timeout
Oct 15, 2026
Merged

thomasfosterau merged 2 commits into
trunkfrom
issue-302-skip-and-timeout

Conversation

@thomasfosterau

@thomasfosterau thomasfosterau commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Implements the owner decisions recorded on #302: a skip list plus a timeout, with no per-rule budget options (2026-10-15 decision on the default and on prefix matching).

Options.skip is a list of rule-id prefixes to leave out, the complement of only.

  • Matching. A prefix is matched exactly as only matches one: a plain startsWith on the rule id. So "spec/links/" covers the three link rules, and "spec/pagination/links-enumerate-collection" covers the pagination walk. Per the owner, plain prefixes stay in this PR. Segment-aware matching for both only and skip (an exact id, or ids that continue it with /) is follow-up feat(Conformance): segment-aware rule-id prefixes for only and skip #367. It cites the spec/status/create-location / spec/status/create-location-matches-self collision and the accept-weight test prefix that would need updating. docs/conformance.md §1.5 links it.
  • How it combines with only. tiers and only select first, and then skip removes rules from that selection (src/Conformance.ts, selected). A rule that both only and skip name is left out.
  • How a skipped rule appears in the Report. It is absent, the same as a rule only does not select. A Skipped outcome keeps its existing meaning: a rule that ran and could not decide.

Options.requestTimeout takes a Duration.Input. It is applied in runner (src/internal/conformance/core.ts).

  • Default: 30 seconds, for verify and verifyWith alike (owner decision). Core.requestTimeoutOf resolves the option, and the runner now takes the resolved Option<Duration> explicitly, with no default parameter of its own.
  • Override. Any positive Duration.Input overrides the default.
  • Off. Turn it off with requestTimeout: "Infinity", or Duration.infinity, which is the same value. "Infinity" is already a Duration.Input, so the option keeps a single type and needs no false sentinel. With the timeout off, the kit waits as long as the transport does, as it did before.
  • Non-positive values are a defect. Zero, a negative duration, NaN (which Duration reads as zero), "-Infinity", or an input Duration cannot read ("1e3 seconds", ".5 seconds", which the type admits) all make the run die with a RangeError before any request is sent. The message reads `requestTimeout` must be a positive duration `Duration` can read, or "Infinity" for no timeout; got ….
  • What it bounds. It covers one request: the client's whole execute, a client's own retry included, plus the reading of the body. Each redirect hop gets a timeout of its own. A request it cuts short is interrupted (over fetch, aborted) and reads as a lost response: status: 0, a transport error naming the timeout, and unfollowed: None. It is therefore never an unfollowed redirect, and the Conformance: follow same-origin redirects (follow-up to #328) #333 skip logic does not fire on it.
  • Tests that relied on a hang.
    • The in-process "stops at an external interrupt" test now sets requestTimeout: "Infinity", because its PATCH must still be waiting when the interrupt comes.
    • The runner's "waits an hour" TestClock test now resolves "Infinity" through requestTimeoutOf.
    • The three examples' in-process verify conformance tests pass unchanged under the default.

Docs.

  • docs/conformance.md §1.5 now covers the default, the off spelling and the defect.
  • §1.5 has a new paragraph: skip, only and a read the timeout cuts short all narrow what Context.observed has seen before the write rules vet a create (write.ts: "only as complete as the reads before it").
  • The error-note example in §1.5 is now a double-backtick code span showing the 30s default message. That restores the five spaces the formatter had dropped around later spans (reachable, execute, retry, execute, cleanup), and Conformance: follow same-origin redirects (follow-up to #328) #333 is now a link.
  • The §3 pagination-walk and link-crawl sentences describe the default bound.
  • §9 describes the new tests.
  • §10's open question about the default is marked answered.
  • README, the Options / verifyWith JSDoc (whose example is now skip: ["spec/links/"]) and the changeset are updated.

Closes #302

Verification

Run in order on head 9bb1a9a (rebased on origin/trunk, which had not moved):

  • pnpm install --frozen-lockfile: pass
  • vp run build: pass
  • vp check: pass (0 errors). This change adds no lint warning: per-rule warning counts on the four changed source and test files match 829a7d0's, and the in-process default test uses Registry.platformLayer rather than a cast.
  • vp run check (sources, type-level tests, examples, emitted declarations): pass
  • vp run -r test:unit: pass (root 3397 tests; examples 43 / 45 / 55)
  • vp run docgen (every JSDoc @example compiled and executed): pass
  • fallow audit --format json --quiet --explain --gate-marker agent: verdict pass
  • fallow audit --base origin/trunk: pass ("No issues in 9 changed files"; 10 inherited findings excluded)
  • pnpm audit --audit-level high: pass (exit 0; 1 moderate advisory, below the threshold)

New tests for the default, the off spelling and the rejection. All run under TestClock or with no clock at all.

  • core.test.ts, requestTimeoutOf:
    • It resolves to 30s when unset and keeps a positive duration.
    • "Infinity" and Duration.infinity resolve to no timeout.
    • It dies for 0, -1, NaN, "-5 seconds", "-Infinity", "1e3 seconds", ".5 seconds" and Duration.zero.
  • core.test.ts, runner (requestTimeout):
    • Under the default, a request is still pending at 29.999s and is lost at 30s.
    • A client's own HttpClient.retry runs inside the timeout. The first attempt hangs and the second would answer, yet under a 1s timeout the result is status: 0 with 1 attempt and 1 interrupt.
    • With "Infinity", the runner waits an hour.
  • Conformance.test.ts, requestTimeout:
    • A run that sets nothing, against a server that never answers, cuts each request short at 30s, with notes naming 30s.
    • The same holds in process: verify against handlers that never answer (Effect.never on get and list) is still pending at 29.999s, cuts the first request short at 30s and the second at 60s, with notes naming 30s.
    • A run with "Infinity" or Duration.infinity is still waiting an hour later.
    • A run with 0, -1, NaN, "-Infinity" or "1e3 seconds" dies before it sends a request. The client fails each request at once, so a run that sent one before reading its options finishes and fails on sent, rather than hanging into vitest's 5s timeout.

Mutation checks. Each mutation was applied (to requestTimeoutOf, or to execute where noted), the core and conformance suites were run, and the mutation was reverted:

Mutation Result
no default (unset resolves to no timeout) 3 tests fail (both default tests and the requestTimeoutOf default)
explicit off ignored ("Infinity" resolves to the default) 5 tests fail (both off tests in each file)
non-positive accepted 10 tests fail (every rejection case in both files)
default applied only to verifyWith (execute: no timeout in process unless set) 1 test fails (the in-process default test)
options read after the rules run (execute: validation moved after the requests) 5 tests fail in milliseconds on expected 2 to be +0 (sent), not on the test timeout

The earlier mutation checks still apply: skip ignored, only overriding skip, no timeout applied, a per-probe timeout, the body read outside the timeout, a timeout classified as an unfollowed redirect, and requestTimeout not passed to runner.

Changeset

  • .changeset/conformance-skip-and-timeout.md: minor, because this is new public API on Conformance.Options. It says in bold that the default changes for every run: a request unanswered after 30s is now a lost response where the run used to wait, and "Infinity" restores the old behaviour.

Notes

  • Follow-up: feat(Conformance): segment-aware rule-id prefixes for only and skip #367, segment-aware prefixes for only and skip.
  • Exported signatures.
    • Conformance.Options gains skip? and requestTimeout?.
    • Internally, Core.requestTimeoutOf and Core.DEFAULT_REQUEST_TIMEOUT are new.
    • The runner's third parameter is now required.
    • Core.ResolvedOptions.requestTimeout is the resolved Option<Duration>.
  • Slow in-process targets. In-process verify runs are now bounded too. A handler slower than 30s per request would read as lost. None of the repo's own suites comes close.
  • What the timeout does not cover. It does not bound cleanup, which is not a request the kit sends. A create it cuts short is treated as having created nothing, though the server may have created it anyway. Both points are documented.

🤖 Generated with Claude Code


Generated by Claude Code

`Options.skip` leaves rules out by id prefix, the complement of `only`
and matched the same way (a plain string prefix; segment-aware matching
for both is #367). It removes rules from what `tiers` and `only` select,
and a rule it leaves out is absent from the report, as one `only` does
not select is.

`Options.requestTimeout` (a `Duration.Input`) bounds each request the
runner sends: the client's whole `execute`, a `retry` included, and the
reading of the body. Each redirect hop has a timeout of its own. A
request it cuts short is interrupted and reads as a lost response,
`status: 0` with a transport error naming the timeout, never as an
unfollowed redirect.

Owner decision on #302: it defaults to thirty seconds for `verify` and
`verifyWith` alike. Any positive duration overrides it; `"Infinity"` (or
`Duration.infinity`) turns it off. A duration that is not positive (zero,
negative, `NaN`) is a defect: the run dies before it sends a request.

Closes #302

Claude-Session: https://claude.ai/code/session_0187g2QpzaHqEzX8GJxiJBVc
Co-Authored-By: Claude <noreply@anthropic.com>
@thomasfosterau
thomasfosterau force-pushed the issue-302-skip-and-timeout branch from 829a7d0 to eb83353 Compare October 15, 2026 06:43
…e durations (#302)

A run of `verify` against handlers that never answer is still waiting a
millisecond before thirty seconds and cuts each request short at thirty,
so the default is pinned for `verify` as well as `verifyWith`; `verify`'s
JSDoc says so, as `verifyWith`'s does.

A `requestTimeout` that `Duration` cannot read (`"1e3 seconds"`,
`".5 seconds"`), which the type admits, dies as a non-positive one does;
the JSDoc, README, design note and changeset now name it, the
`requestTimeoutOf` table and the run-level test cover it, and the message
asks for "a positive duration `Duration` can read".

The run-level "dies before it sends a request" tests now use a client that
fails at once, so a run that read its options after sending would finish
and fail on `sent`, not hang into the test timeout.

Part of #302

Claude-Session: https://claude.ai/code/session_0187g2QpzaHqEzX8GJxiJBVc
Co-Authored-By: Claude <noreply@anthropic.com>
@thomasfosterau
thomasfosterau merged commit b81012b into trunk Oct 15, 2026
1 check passed
@thomasfosterau
thomasfosterau deleted the issue-302-skip-and-timeout branch October 15, 2026 08:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(Conformance): a way to opt out of, or bound, the pagination walk

2 participants