From eb8335353bba09f5659d48e4c365682fafef83ba Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 15 Oct 2026 06:40:41 +0000 Subject: [PATCH 1/2] feat(Conformance): `skip` and `requestTimeout` options (#302) `Options.skip` leaves rules out by id prefix, the complement of `only` and matched the same way (a plain string prefix; segment-aware matching for both is #367). It removes rules from what `tiers` and `only` select, and a rule it leaves out is absent from the report, as one `only` does not select is. `Options.requestTimeout` (a `Duration.Input`) bounds each request the runner sends: the client's whole `execute`, a `retry` included, and the reading of the body. Each redirect hop has a timeout of its own. A request it cuts short is interrupted and reads as a lost response, `status: 0` with a transport error naming the timeout, never as an unfollowed redirect. Owner decision on #302: it defaults to thirty seconds for `verify` and `verifyWith` alike. Any positive duration overrides it; `"Infinity"` (or `Duration.infinity`) turns it off. A duration that is not positive (zero, negative, `NaN`) is a defect: the run dies before it sends a request. Closes #302 Claude-Session: https://claude.ai/code/session_0187g2QpzaHqEzX8GJxiJBVc Co-Authored-By: Claude --- .changeset/conformance-skip-and-timeout.md | 5 + README.md | 5 + docs/conformance.md | 32 ++- src/Conformance.test.ts | 252 +++++++++++++++++++-- src/Conformance.ts | 86 +++++-- src/internal/conformance/core.test.ts | 235 ++++++++++++++++++- src/internal/conformance/core.ts | 136 ++++++++++- src/internal/conformance/write.test.ts | 1 + 8 files changed, 698 insertions(+), 54 deletions(-) create mode 100644 .changeset/conformance-skip-and-timeout.md diff --git a/.changeset/conformance-skip-and-timeout.md b/.changeset/conformance-skip-and-timeout.md new file mode 100644 index 00000000..fc8dbc2f --- /dev/null +++ b/.changeset/conformance-skip-and-timeout.md @@ -0,0 +1,5 @@ +--- +"@thomasfosterau/effect-jsonapi": minor +--- + +**`Conformance.Options` gains `skip` and `requestTimeout`, two ways to bound a run against a live server (#302).** `skip` is a list of rule-id prefixes to leave out, the complement of `only` and matched the same way (a plain string prefix of the id): `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl. It removes rules from what `tiers` and `only` select, so a rule both `only` and `skip` name is left out, and a rule it leaves out is absent from the report, as one `only` does not select is. `requestTimeout` takes any positive `Duration.Input` and bounds each request the kit sends — the client's whole `execute`, a `retry` it wraps itself in included, and the reading of the body — with each redirect hop a request of its own. A request still unanswered then is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout, which every rule treats as any other transport error and never as a redirect the kit did not follow. **This changes the default for every run:** `requestTimeout` defaults to thirty seconds, for `verify` and `verifyWith` alike, so a run that sets nothing now reads a request unanswered after thirty seconds as a lost response where it used to wait for as long as the transport did. `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off and restores the old behaviour; a duration that is not positive (zero, negative, `NaN`) is a defect. `skip` is unset by default, leaving out nothing. diff --git a/README.md b/README.md index e9da600a..0980ea5f 100644 --- a/README.md +++ b/README.md @@ -2003,6 +2003,11 @@ Two tiers, separately runnable. `spec` is JSON:API v1.1 alone; `profile` is this Use `Conformance.assert(report)` as a gate: it fails with a `ContractViolation` carrying the rule, its requirement, its citation and the request that decided it. A rule that died is not a finding about your server, but the run could not decide it either, so `assert` dies on it — with a defect tagged `RuleDied` naming the rule and the defect, the original as its `cause` — rather than passing. If you gate on `summarise(report).failed` or `findings(report)` instead of `assert`, also check `summarise(report).errored === 0`. +**Against a live server, bound the run.** `verifyWith` runs the identical rules over any `HttpClient` and `baseUrl`. Two options bound what that costs the server: + +- `skip` — rule-id prefixes to leave out, the complement of `only` and matched the same way: `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl, the rules that make the most requests. It removes rules from what `tiers` and `only` select, and a rule it leaves out is absent from the report, as one `only` does not select is. +- `requestTimeout` — a `Duration.Input` (`"10 seconds"`) bounding each request, its body included; each redirect hop gets its own. A request still unanswered then is aborted and reads as a lost response naming the timeout, which every rule treats as any transport error. Thirty seconds by default, for `verify` and `verifyWith` alike; `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off, so the kit waits as long as the transport does. A duration that is not positive is a defect. + Full design note, including the coverage ledger and what it would take to close each gap: [`docs/conformance.md`](./docs/conformance.md). ## Examples diff --git a/docs/conformance.md b/docs/conformance.md index 7c4620cc..6e59fef1 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -109,6 +109,28 @@ Four edges of the contract, stated so nobody relies on the opposite: **`cleanup` is optional.** Against a target known to be disposable — an in-process store the test throws away — it can be omitted; the kit then leaves what it created in place and says so (`left`, naming `writes[""].cleanup`). Against a server whose state outlives the run, always supply it. This is the less demanding of the two contracts, chosen pending the owner's decision ([§10](#10-open-questions)); the alternative is to require `cleanup` whenever `create` is given. +### 1.5 Against a live server: `skip` and `requestTimeout` + +Two options bound what a run costs a server whose state and uptime matter (owner decision on [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)): which rules run, and how long the kit waits for an answer. + +```ts +Conformance.verifyWith(api, { + client, + baseUrl: "https://api.example.com", + ids: { articles: "1" }, + requestTimeout: "10 seconds", // any Duration.Input; thirty seconds when unset + skip: ["spec/pagination/links-enumerate-collection", "spec/links/"], +}); +``` + +**`skip` leaves rules out by id prefix — the complement of `only`.** A prefix is matched exactly as `only` matches one, as a plain string prefix of the rule id, so `"spec/links/"` leaves out the three link rules (the link crawl, §3) and `"spec/pagination/links-enumerate-collection"` the pagination walk, the kit's most expensive rules against a live server. The three selections compose in one order: `tiers` and `only` select, and `skip` removes from what they selected, so a rule both `only` and `skip` name is left out. A rule `skip` leaves out is absent from the report, exactly as one `only` does not select is: the report lists what the run tried, and a `Skipped` outcome means a rule that ran and could not decide (§4.1), not one the caller declined to run. A prefix that matches nothing leaves nothing out, as one in `only` selects nothing. A plain string prefix can reach past the end of a segment: `"spec/status/create-location"` also matches `spec/status/create-location-matches-self`. Making both `only` and `skip` match by segment (an exact id, or ids that continue it with `/`) is [#367](https://github.com/thomasfosterau/effect-jsonapi/issues/367); until then, end a section's prefix with `/`. + +**What a run leaves out narrows what the write rules vet a create against.** A create is refused when it names an id a successful `GET` earlier in the run has shown (`Context.observed`, §1.4), a check only as complete as the reads before it — so a read rule `skip` or `only` leaves out, or a read `requestTimeout` cuts short, is a read that check does not see. + +**`requestTimeout` bounds each request in time.** A request still unanswered when it runs out is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout (``no response (no answer within 30s, the run's `requestTimeout`)``). Every rule treats it exactly as it treats a refused connection or a reset — `spec/target/reachable` reports it, the pagination walk and the link crawl leave what it cut short not decided — and never as a redirect the kit did not follow, so the [#333](https://github.com/thomasfosterau/effect-jsonapi/pull/333#issuecomment-6181006987) rule that a request a redirect left unanswered skips the rule that sent it does not apply. It bounds one request: the client's whole `execute`, a `retry` the client wraps itself in included (the kit still runs each request through the client exactly as its `execute` does, §1.2), and the reading of the answer's body, so a server that sends its headers and holds the body open is cut short too. Each redirect hop the kit follows is a request of its own, with a timeout of its own and counting against the rule's budget as before, so one probe waits at most six timeouts (the first request and five hops). A request it cuts short was still sent, and still counts against the budget. A create it cuts short is taken to have created nothing, as a non-2xx one is (§1.4): the server may have created it all the same. It does not bound `cleanup`, which is not a request the kit sends (§1.4). + +**It defaults to thirty seconds**, for `verify` and `verifyWith` alike (owner decision on [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302#issuecomment-6186007738)), so a server that holds a response open costs a run at most that per request, and a run that sets nothing is bounded too. Any positive `Duration.Input` overrides it. `requestTimeout: "Infinity"` (or `Duration.infinity`, the same value) turns it off: the kit then waits as long as the transport does, as it did before the option existed, so a server that holds a response open holds the run. `Infinity` is the spelling because it is already a `Duration.Input`, so the option keeps one type and needs no second sentinel. A duration that is not positive — zero, a negative one, or `NaN`, which `Duration` reads as zero — would cut every request short before it was answered, so it is a defect in the options: the run dies before it sends a request, rather than reporting every rule undecided. + ## 2. Two tiers, separately runnable | Tier | What it is | Authority | @@ -154,7 +176,7 @@ Whether an endpoint declares the family is read from discovery (§1: its flat wi **Why a `next` to an empty page is a MUST failure.** The base specification defines `next` as "the next page of data" and requires an unavailable link to be omitted or `null`. A page that lists no resources is not a next page of data: the page before it was the last, so its `next` was a link to a page that does not exist, where the spec required none. The [cursor-pagination profile](https://jsonapi.org/profiles/ethanresnick/cursor-pagination/) does allow such a link, but not on a page the walk reads `next` from. Its §Links holds a server to the same answer for any request without `page[before]`: "If a request does not contain the `page[before]` parameter, the server **MUST** determine whether a next page exists, and return `null` as the `next` link if not." The rest — a request that carries `page[before]` — falls to the sentences after it: "In all other cases, a server **SHOULD** set these links to `null` when it can inexpensively determine that the current response is for the first or last page respectively", and "if the server can't easily determine whether there are … subsequent results (when computing the `next` link), it **MAY** use a URI in these links that returns an empty array as its paginated data." The kit never adds `page[before]`: its first request sets only a page size, and every later request is a `next` link exactly as the server wrote it, which the profile's own examples spell with `page[after]` alone. So every page the walk reads `next` from falls under the profile's MUST, and a `next` there to an empty page fails under the profile as it does under the base specification. A server whose own `next` link carried `page[before]` would earn the MAY on the page it leads to; the kit does not recognise that case, and reads `next` as the base specification defines it. A page whose `data` holds items but none with a string `type` and `id` (or `lid`) is not taken for an empty page. The kit cannot say what such a page lists, so the walk is reported not decided. The page is not counted as empty. A walk that would start on such a page — the reference or the second walk — is not begun, and the collection is reported not decided, since nothing such a walk found could be held against resources the kit never saw. -The walk is bounded in requests, so a hostile or broken pager cannot keep the kit walking forever or make the rule die: 20 pages per walk, 200 requests per run of the rule (the `first` / `last` checks and the order recheck below included), and a `next` that leads back to a visited page, or to a page adding no resource not already seen, ends it with a finding. It is not bounded in time: there is no per-request timeout, so a server that holds one response open holds the run with it, as for every other rule (a timeout is a candidate for [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). A collection the budget runs out on, at any of those requests, is reported not decided, naming the budget. It is never a partial pass. A server error, a `429`, a `408` (RFC 9110 §15.5.9: the server "did not receive a complete request message within the time that it was prepared to wait") or a lost response mid-walk, or from a `first` / `last` link, says nothing of the links, so it leaves the collection not decided, as it does for the reference request. A `next` that leads to any other 4xx, or to a 2xx that is not a page, is a finding. A link resolves against the page that carries it, and one that leads to another origin is not followed (`Skipped`, naming the origin). Server values in findings render as `resourceLabel` and `Formatter.format` render them ([#284](https://github.com/thomasfosterau/effect-jsonapi/pull/284)), so a link that is `{"toString":1}` is a finding, not a dead rule. +The walk is bounded in requests, so a hostile or broken pager cannot keep the kit walking forever or make the rule die: 20 pages per walk, 200 requests per run of the rule (the `first` / `last` checks and the order recheck below included), and a `next` that leads back to a visited page, or to a page adding no resource not already seen, ends it with a finding. It is bounded in time by `requestTimeout` (§1.5, thirty seconds unless the run says otherwise), which bounds each of those requests; a run that turns it off is held by a server that holds one response open, as for every other rule. A request the timeout cuts short is a lost response, so it leaves the collection not decided. `skip: ["spec/pagination/links-enumerate-collection"]` leaves the walk out altogether (§1.5). A collection the budget runs out on, at any of those requests, is reported not decided, naming the budget. It is never a partial pass. A server error, a `429`, a `408` (RFC 9110 §15.5.9: the server "did not receive a complete request message within the time that it was prepared to wait") or a lost response mid-walk, or from a `first` / `last` link, says nothing of the links, so it leaves the collection not decided, as it does for the reference request. A `next` that leads to any other 4xx, or to a 2xx that is not a page, is a finding. A link resolves against the page that carries it, and one that leads to another origin is not followed (`Skipped`, naming the origin). Server values in findings render as `resourceLabel` and `Formatter.format` render them ([#284](https://github.com/thomasfosterau/effect-jsonapi/pull/284)), so a link that is `{"toString":1}` is a finding, not a dead rule. **An unstable order is not a violation of the links.** §Sorting: "The server **MAY** apply default sorting rules to top-level `data` if request parameter `sort` is not specified" — without `sort`, JSON:API requires no stable order, so offset pagination over rows that shift between requests may repeat or skip a resource without any link being wrong. So when the walk finds a problem that depends on order (a resource repeated or never reached, a `first` / `last` page that disagrees), the kit asks for each walk's first page again; if one comes back in a different order, those findings are withheld and the collection is reported not decided, quoting §Sorting. A recheck the budget leaves no request for, or that gets no page back, confirms nothing, so it withholds them too. This holds on every path out of the rule, including a walk cut short. The order-independent findings — `prev` on the first page, a `next` that cycles, dangles or is not a link — stand either way. Over an order that two identical requests reproduce, a repeat or a gap means `next` was not "the next page of data", so the rule is a MUST, the spec's own level. A server whose collection changes during the walk (a live server under writes) can show the same symptoms; the finding names the pages so they can be compared. @@ -179,7 +201,7 @@ The links are collected from one `GET` of every read endpoint whose URL resolves **What the kit cannot identify, it does not decide on, on either side.** An answer listing an item without a string `type` and `id` leaves the link not decided. So does a claim the kit cannot read: a resource object with no string `type` and `id` cannot have its `self` held to it, and linkage naming such an item cannot have its `related` held to it. Such a link is noted as not decided, without a request, rather than dropped (the `self` case) or compared with the unidentified item left out (the linkage case, which would report a resource the linkage does name as one it does not). -The crawl is the pagination walk's (`src/internal/conformance/crawl.ts`, shared): `GET` only, and a link that leads to another origin (scheme, host or port) is never requested and is reported not followed; one whose scheme gives it no origin (`javascript:`, `data:`) is named by its scheme. A redirect is followed only within the base URL's origin (§1.2), each hop counting against the budget, and the answer it leads to is what the link is held to; one the kit does not follow — off the origin, past five hops — leaves the link not decided and the rule `Skipped`, naming its `Location` (§1.2), so a link cannot reach another origin that way either. A server error, a `429`, a `408` or a lost response leaves that link not decided, never failed. So does a `401` or `403`: access control is outside JSON:API, so a link the kit may not follow is not a broken one. Any other non-2xx — a `404`, or a `3xx` with no `Location` to follow — is a finding: the run has just received the link from the server, so a 404 for it is the server contradicting itself (a live server under writes can legitimately delete a resource between the two requests; the finding names both, so they can be compared). Each rule may make 200 requests per run, the collecting `GET`s included. A crawl the budget cannot finish — a read endpoint it had no request left to search, or a link it had none left to follow — is `Skipped`, naming the budget and how far it got, never a partial pass, unless a link it did follow already broke the rule, which stands. Like the pagination walk, the crawl is bounded in requests, not in time: there is no per-request timeout, so a server that holds one response open holds the run with it (a timeout is a candidate for [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). A rule that decided some links and not others passes with a note naming each it could not decide; with none decided it skips, saying why. With no link to follow it skips, naming the endpoints it searched and how each that did not answer 2xx answered. A failed rule lists its first ten findings and one more counting the rest, and every server value a finding names — a resource's `type:id` included — is quoted cut to 200 UTF-16 code units, so a server with a broken link on every resource, or an id a megabyte long, does not make a report of the same size. +The crawl is the pagination walk's (`src/internal/conformance/crawl.ts`, shared): `GET` only, and a link that leads to another origin (scheme, host or port) is never requested and is reported not followed; one whose scheme gives it no origin (`javascript:`, `data:`) is named by its scheme. A redirect is followed only within the base URL's origin (§1.2), each hop counting against the budget, and the answer it leads to is what the link is held to; one the kit does not follow — off the origin, past five hops — leaves the link not decided and the rule `Skipped`, naming its `Location` (§1.2), so a link cannot reach another origin that way either. A server error, a `429`, a `408` or a lost response leaves that link not decided, never failed. So does a `401` or `403`: access control is outside JSON:API, so a link the kit may not follow is not a broken one. Any other non-2xx — a `404`, or a `3xx` with no `Location` to follow — is a finding: the run has just received the link from the server, so a 404 for it is the server contradicting itself (a live server under writes can legitimately delete a resource between the two requests; the finding names both, so they can be compared). Each rule may make 200 requests per run, the collecting `GET`s included. A crawl the budget cannot finish — a read endpoint it had no request left to search, or a link it had none left to follow — is `Skipped`, naming the budget and how far it got, never a partial pass, unless a link it did follow already broke the rule, which stands. Like the pagination walk, the crawl is bounded in requests, and in time by `requestTimeout` (§1.5) unless the run turns it off; a request the timeout cuts short is a lost response, which leaves its link not decided. `skip: ["spec/links/"]` leaves the crawl out altogether (§1.5). A rule that decided some links and not others passes with a note naming each it could not decide; with none decided it skips, saying why. With no link to follow it skips, naming the endpoints it searched and how each that did not answer 2xx answered. A failed rule lists its first ten findings and one more counting the rest, and every server value a finding names — a resource's `type:id` included — is quoted cut to 200 UTF-16 code units, so a server with a broken link on every resource, or an id a megabyte long, does not make a report of the same size. **Status codes** — 404 for an absent resource. With the write fixture (§1.4): a creation without a client-generated id answers 201 with the created resource as primary data (`create-response-document`); the 201 carries `Location` (`create-location`, a SHOULD — the spec's level, corrected from the MUST the never-run v1 rule claimed) and, when the resource also carries `links.self`, the two match (`create-location-matches-self`); an update answers 202, 200 with a document whose primary data — when it has any — is the updated resource (JSON:API 1.1 lets a 200 carry no primary data when the server changed nothing beyond the request; the document must then still be one, so the kit wants a top-level `meta` and no `errors`), or 204 with no response document (`update-response`); and a delete answers 204 empty, 200 with only `meta`, or 202 (`delete-response`). @@ -329,11 +351,13 @@ A kit that fails everything catches every bug and is worthless. The control — **The link rules get their own server** (`linked` in `Conformance.test.ts`), because the main fixture's resources carry no links: two articles, each with a `self`, a to-one `author`, a to-many `tags` and a links-only `comments`, every link resolving to what it names. Against it all three rules pass, with the requests asserted, so a pass is shown to have followed each link once (a link to an endpoint already fetched costs nothing; `related-matches-linkage` does not follow the links-only `comments`). Then each way a link breaks, one at a time, with the finding pinned: a `self` that leads to another resource, to a 404 or to `data: null`; a `related` that answers something other than resource data (`related-dereferences`); and a `related` that omits a resource its linkage names, answers one it does not name, answers another to-one resource, or answers a resource where the linkage is `null` (`related-matches-linkage`, while `related-dereferences` passes the same answer). A `related` that answers 404 or is not a link at all fails `related-dereferences` once and is left not decided by `related-matches-linkage`, which skips when no relationship carries linkage. The subset relaxations are pinned both ways — a page of the related collection, and a page of linkage, each passing when it is consistent and failing when it is not — and two pages are not compared; nor are an answer, a resource or linkage holding an item without a string `type` and `id`. Link objects, a relative link (one that resolves differently against the document and against the base URL) and an absolute link to the run's own origin are followed; a link that is `{"toString":1}` is a finding; a link to another origin is never requested (every request is asserted to go to the run's origin) and is noted, or skipped when it is the only one; a `javascript:` link is named by its scheme. A `fetch` that would follow a redirect off the origin is never let to, even when the run's own `RequestInit` says `redirect: "follow"`, and the rule is skipped, naming the `Location`, never passed on the links that did decide; an `http` → `https` redirect on the same host is treated the same way, and so is a read endpoint redirected off the origin while every link it did find decides. A redirect within the origin is followed, its final answer graded (a different resource there is a finding) and its hop counted; a chain of five hops is followed and one of six skips the rule; a relative link in a redirected answer resolves against where the answer came from; and an answer a transport fetched from another origin all the same is not read, and skips the rule, naming where the transport ended up, rather than passing it on the links that did decide. A `related` answering `{"data":["x"]}`, `{"data":"x"}` or `{"data":[{"id":"1"}]}` is a finding; `{"data":null}` passes. A label the linkage names is quoted cut like any other. A 503, a 500, a 429, a 408, a lost response and a 403 leave the link not decided. The budget is pinned at exactly 200 requests over 250 articles (skipped, naming it), and over 201 read endpoints, one never searched (skipped, naming it, and not counting a read endpoint whose URL does not resolve as searched); a finding made before it ran out still fails the rule. Twenty-five broken links list ten findings and count the rest, and a 300-character id is quoted cut. The roles searched are pinned: a related endpoint's answer is, a relationship endpoint is never requested. The skips name what they searched. An in-process case holds the real router to the rule, including a link to a path no route serves, which fails it rather than killing it; `src/internal/conformance/core.test.ts` pins that only `RouteNotFound` is answered so, any other router failure dying. +**`skip` and `requestTimeout` are pinned at both ends** ([#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). `Conformance.test.ts` asserts that `skip` leaves out exactly the rules its prefixes match (the pagination walk, the link crawl and a section among them), adding no `Skipped` outcome; that it removes rules from what `only` and `tiers` select; and that a rule both `only` and `skip` name is left out. Against a canned server that never answers, a run with a 10ms `requestTimeout` finishes, interrupts every request it sent, and reports exactly what a transport that fails at once reports, but for the note naming the timeout; against the conformant server, a timeout no request reaches changes no outcome, through a client's `retry` too. `core.test.ts` pins the runner under the test clock: a request is still pending a millisecond before the timeout and a lost response naming it at the timeout, with no redirect recorded; a body held open is cut short too; each redirect hop has a timeout of its own (two 600ms hops pass under a one-second timeout, and a hop that never answers is cut short a second after it is requested, counting as a request sent); a client's own `retry` runs within the timeout (a first attempt that hangs is cut short with the run's one request, and the second attempt, which would answer, is never made); and with the timeout turned off (`"Infinity"`) the runner waits as long as the transport does. The default is pinned twice: `core.test.ts` holds `requestTimeoutOf` to thirty seconds when unset, no timeout for `"Infinity"` and `Duration.infinity`, and a defect for zero, a negative duration and `NaN`, and the runner to a request still pending a millisecond before thirty seconds; `Conformance.test.ts` holds a run that sets nothing to cutting each request short at thirty seconds under the test clock, a run that turns it off to still waiting an hour later, and a run with a non-positive timeout to dying before it sends a request. + ## 10. Open questions - **Is `cleanup` required?** The write fixture ([#162](https://github.com/thomasfosterau/effect-jsonapi/issues/162)) ships it **optional** — the less demanding contract — so an in-process target known to be disposable can omit it, and the kit reports what it left (§1.4). The alternative is to require `cleanup` whenever `create` is given, which makes a live-server run safe by construction at the cost of a no-op cleanup in every in-process test. An owner decision. - **Machine-readable output.** `format` renders text. A JSON rendering of `Report` is trivially derivable (it is plain data) but no stable serialisation is committed to yet. -- **Opting out of the pagination walk.** It is the kit's most expensive rule — six to ten requests per paginated collection, at most 200 per run — and it walks a live server's collections. It ships with no dedicated switch: `only` selects rules, so excluding it means listing the others. Whether `Options` should gain an opt-out (or a budget) is open ([#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). +- **A default `requestTimeout`** — answered: thirty seconds, for `verify` and `verifyWith` alike, overridable and with `"Infinity"` to turn it off (§1.5; owner decision on [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302#issuecomment-6186007738)). The cost it accepts is that an endpoint slower than that reads as a lost response; a run against one sets a longer timeout. - **A link the same run just received that answers 404.** The link rules ([#168](https://github.com/thomasfosterau/effect-jsonapi/issues/168)) report it as a finding, and leave a `401` / `403` not decided. #168 left this to be decided; the alternative is to treat a 404 as not decided too, which would excuse exactly the drift (a moved route) the rules exist to catch. An owner decision. -- **Bounding the link crawl.** Each link rule makes at most 200 requests per run and skips when that is not enough, so a server whose read endpoints answer large collections by default gets these rules skipped rather than sampled. Whether to sample (some links per endpoint) or make the budget an option is the same question as the pagination walk's ([#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). +- **Sampling the link crawl.** Each link rule makes at most 200 requests per run and skips when that is not enough, so a server whose read endpoints answer large collections by default gets these rules skipped rather than sampled. [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302) settled the cost question with `skip` and `requestTimeout` (§1.5) and no per-rule budget option; sampling some links per endpoint remains possible later. - **Rule versioning.** Rule ids are the stable surface a consumer pins against; a policy for renaming or retiring one is not yet written. diff --git a/src/Conformance.test.ts b/src/Conformance.test.ts index c133f500..5cefd9fb 100644 --- a/src/Conformance.test.ts +++ b/src/Conformance.test.ts @@ -11,6 +11,7 @@ import { Array as Arr, Cause, Deferred, + Duration, Effect, Exit, Fiber, @@ -24,6 +25,7 @@ import { import { FetchHttpClient, HttpClient, HttpClientError, HttpClientResponse } from "effect/http"; import { HttpClientRequest } from "effect/http"; import { HttpApi, HttpApiBuilder, HttpApiEndpoint, HttpApiGroup } from "effect/http-api"; +import { TestClock } from "effect/testing"; import * as Conformance from "./Conformance.js"; import * as Endpoint from "./Endpoint.js"; import * as Filter from "./Filter.js"; @@ -721,6 +723,47 @@ describe("Conformance: discovery and selection", () => { if (skipped._tag === "Skipped") expect(skipped.reason).toContain(`ids["articles"]`); }); + const ids = (report: Conformance.Report) => report.outcomes.map((outcome) => outcome.rule.id); + + it("leaves out the skipped rule ids, matching on prefix as `only` does", async () => { + const all = ids(await verifyCanned(conformantServe)); + const skip = ["spec/pagination/links-enumerate-collection", "spec/links/", "spec/content-neg"]; + const report = await verifyCanned(conformantServe, { skip }); + const left = all.filter((id) => !skip.some((prefix) => id.startsWith(prefix))); + // Each prefix leaves something out, and nothing else is. + expect(skip.every((prefix) => all.some((id) => id.startsWith(prefix)))).toBe(true); + expect(left.length).toBeLessThan(all.length - 5); + expect(ids(report)).toEqual(left); + // A skipped rule is absent from the report, as one `only` did not select + // is — never a `Skipped` outcome that would read as a rule the run tried. + expect(Conformance.summarise(report).skipped).toBe( + Conformance.summarise(await verifyCanned(conformantServe, { only: left })).skipped, + ); + }); + + it("leaves out the skipped rules from what `only` and `tiers` select", async () => { + const selected = ids( + await verifyCanned(conformantServe, { only: ["spec/content-negotiation"] }), + ); + const weight = "spec/content-negotiation/accept-weight"; + const report = await verifyCanned(conformantServe, { + only: ["spec/content-negotiation"], + skip: [weight], + }); + expect(selected.filter((id) => id.startsWith(weight)).length).toBeGreaterThan(1); + expect(ids(report)).toEqual(selected.filter((id) => !id.startsWith(weight))); + // A rule both name is left out; `skip` naming everything leaves nothing. + const both = await verifyCanned(conformantServe, { + only: ["spec/document/top-level-members"], + skip: ["spec/document/top-level-members"], + }); + expect(both.outcomes).toEqual([]); + const profile = await verifyCanned(conformantServe, { tiers: ["profile"], skip: ["spec/"] }); + expect(profile.outcomes.length).toBeGreaterThan(0); + const spec = await verifyCanned(conformantServe, { tiers: ["spec"], skip: ["spec/"] }); + expect(spec.outcomes).toEqual([]); + }); + it("records the transport it drove", async () => { const report = await verifyCanned(conformantServe); expect(report.transport).toBe("http"); @@ -728,6 +771,165 @@ describe("Conformance: discovery and selection", () => { }); }); +describe("Conformance: requestTimeout", () => { + /** A client that never answers, counting the requests the timeout cut short. */ + const silent = () => { + let interrupted = 0; + const client = HttpClient.make(() => + Effect.never.pipe( + Effect.onInterrupt(() => + Effect.sync(() => { + interrupted += 1; + }), + ), + ), + ); + return { client: () => client, interrupted: () => interrupted }; + }; + + const rules = ["spec/target/reachable", "spec/content-negotiation/response-media-type"]; + + it("reads a server that never answers as a lost response, rather than hanging", async () => { + const server = silent(); + const report = await verifyCanned( + conformantServe, + { requestTimeout: 10, only: rules }, + server.client, + ); + const note = "no answer within 10ms, the run's `requestTimeout`"; + expect(outcomeFor(report, "spec/target/reachable")).toMatchObject({ + _tag: "Failed", + findings: [ + { detail: "/articles/1 produced no HTTP response", observed: `no response (${note})` }, + { detail: "/articles produced no HTTP response", observed: `no response (${note})` }, + ], + }); + // Every request was cut short, not left running. + expect(server.interrupted()).toBe(Conformance.summarise(report).probes); + // Exactly what a transport that fails at once reports, but for the note: a + // timeout is a transport error, never a redirect the kit did not follow. + const lost = await verifyCanned(conformantServe, { only: rules }, () => + HttpClient.make((request) => + Effect.fail( + new HttpClientError.HttpClientError({ + reason: new HttpClientError.TransportError({ request, description: "reset" }), + }), + ), + ), + ); + const why = (outcomes: ReadonlyArray) => + Arr.map(outcomes, (outcome) => + Predicate.isTagged(outcome, "Failed") + ? { + ...outcome, + findings: Arr.map(outcome.findings, (found) => ({ + ...found, + observed: found.observed.replace(/^no response \(.*\)$/, "no response (…)"), + })), + } + : outcome, + ); + expect(why(report.outcomes)).toEqual(why(lost.outcomes)); + }); + + /** + * A run of `reachable` against a server that never answers, under the test + * clock, advanced by each of `steps` in turn; `settled` says, after each + * step, whether the run had finished. A run still going after the last step + * is interrupted. + */ + const clocked = (options: Conformance.Options, steps: ReadonlyArray) => { + const server = silent(); + return Effect.runPromiseExit( + Effect.gen(function* () { + const fiber = yield* Effect.forkChild( + Conformance.verifyWith(api, { + client: server.client(), + baseUrl: "http://fixture.test", + ids: { articles: "1" }, + only: ["spec/target/reachable"], + ...options, + }), + ); + const settled = yield* Effect.forEach( + steps, + (step) => Effect.map(TestClock.adjust(step), () => fiber.pollUnsafe() !== undefined), + { concurrency: 1 }, + ); + const report = + settled.at(-1) === true ? Option.some(yield* Fiber.join(fiber)) : Option.none(); + yield* Fiber.interrupt(fiber); + return { settled, report, interrupted: server.interrupted() }; + }).pipe(Effect.provide(TestClock.layer())), + ); + }; + + it("waits thirty seconds for each request by default", async () => { + const exit = await clocked({}, ["29999 millis", "1 millis", "30 seconds"]); + expect(Exit.isSuccess(exit)).toBe(true); + if (Exit.isFailure(exit)) return; + const run = exit.value; + // The first request is cut short at thirty seconds, the second thirty after. + expect(run.settled).toEqual([false, false, true]); + expect(run.interrupted).toBe(2); + const note = "no answer within 30s, the run's `requestTimeout`"; + expect(Option.map(run.report, (report) => outcomeFor(report, "spec/target/reachable"))).toEqual( + Option.some( + expect.objectContaining({ + _tag: "Failed", + findings: [ + expect.objectContaining({ observed: `no response (${note})` }), + expect.objectContaining({ observed: `no response (${note})` }), + ], + }), + ), + ); + }); + + it.each(["Infinity", Duration.infinity] as const)( + "waits as long as the transport does when the run turns it off with %s", + async (requestTimeout) => { + const exit = await clocked({ requestTimeout }, ["30 seconds", "1 hour"]); + expect(Exit.isSuccess(exit)).toBe(true); + if (Exit.isFailure(exit)) return; + const run = exit.value; + expect(run.settled).toEqual([false, false]); + // Only the first request was sent; it was still waiting. + expect(run.interrupted).toBe(1); + }, + ); + + it.each([0, -1, Number.NaN, "-Infinity"] as const)( + "dies on a `requestTimeout` of %s before it sends a request", + async (requestTimeout) => { + let sent = 0; + const exit = await Effect.runPromiseExit( + Conformance.verifyWith(api, { + client: HttpClient.make(() => + Effect.sync(() => { + sent += 1; + }).pipe(Effect.andThen(Effect.never)), + ), + baseUrl: "http://fixture.test", + ids: { articles: "1" }, + requestTimeout, + }), + ); + expect(Exit.isFailure(exit) && Cause.hasDies(exit.cause)).toBe(true); + expect(Exit.isFailure(exit) && String(Cause.squash(exit.cause))).toContain( + "`requestTimeout` must be a positive duration", + ); + expect(sent).toBe(0); + }, + ); + + it("changes nothing for a server that answers within it", async () => { + const report = await verifyCanned(conformantServe, { requestTimeout: "1 minute" }); + expect(failed(report)).toEqual([]); + expect(report.outcomes).toEqual((await verifyCanned(conformantServe)).outcomes); + }); +}); + // --------------------------------------------------------------------------- // The control: a conformant server // --------------------------------------------------------------------------- @@ -2842,26 +3044,34 @@ describe("Conformance: the write fixture", () => { expect(report.outcomes.filter((outcome) => Predicate.isTagged(outcome, "Errored"))).toEqual([]); }); - it("lets a client retry a request whose preparation failed, as its own `execute` does", async () => { - const direct = await run(); - const failedOnce = new WeakSet(); - let refused = 0; - const { report } = await run(undefined, undefined, { - client: (canned) => - HttpClient.retry( - HttpClient.mapRequestEffect(canned, (request) => { - if (failedOnce.has(request)) return Effect.succeed(request); - failedOnce.add(request); - refused += 1; - return Effect.fail(unprepared(request)); - }), - { times: 1 }, - ), - }); - expect(refused).toBeGreaterThan(0); - expect(failed(report)).toEqual([]); - expect(verdicts(report)).toEqual(verdicts(direct.report)); - }); + // The timeout wraps the client's whole `execute`, so a `retry` runs within it + // and the request still runs through the client exactly as before. + it.each([ + ["", undefined], + [", within a `requestTimeout`", "1 minute"], + ] as const)( + "lets a client retry a request whose preparation failed, as its own `execute` does%s", + async (_, requestTimeout) => { + const direct = await run(); + const failedOnce = new WeakSet(); + let refused = 0; + const { report } = await run(undefined, () => ({ requestTimeout }), { + client: (canned) => + HttpClient.retry( + HttpClient.mapRequestEffect(canned, (request) => { + if (failedOnce.has(request)) return Effect.succeed(request); + failedOnce.add(request); + refused += 1; + return Effect.fail(unprepared(request)); + }), + { times: 1 }, + ), + }); + expect(refused).toBeGreaterThan(0); + expect(failed(report)).toEqual([]); + expect(verdicts(report)).toEqual(verdicts(direct.report)); + }, + ); it("names a redirect the transport followed off the origin in the rule it leaves undecided", async () => { const unknown = await run(undefined, undefined, { @@ -3222,6 +3432,8 @@ describe("Conformance: the write fixture", () => { ids: { articles: "1" }, mutating: true, writes: { articles: fixture(store, cleaned) }, + // The PATCH must still be waiting when the interrupt comes. + requestTimeout: "Infinity", }), ); yield* Deferred.await(patching); diff --git a/src/Conformance.ts b/src/Conformance.ts index 1cf3d128..665e05da 100644 --- a/src/Conformance.ts +++ b/src/Conformance.ts @@ -90,6 +90,7 @@ import { Array as Arr, Data, + Duration, Effect, Layer, MutableHashSet, @@ -757,33 +758,72 @@ export interface Options { readonly tiers?: ReadonlyArray | undefined; /** * Run only these rule ids. A prefix matches every rule beneath it, so - * `["spec/content-negotiation"]` selects that whole section. + * `["spec/content-negotiation"]` selects that whole section. A rule it does + * not select is left out of the run and out of the report. */ readonly only?: ReadonlyArray | undefined; + /** + * Leave out these rule ids — the complement of `only`, matched the same + * way: a prefix (a plain string prefix of the id) matches every rule + * beneath it, so `["spec/links/"]` leaves out the link crawl and + * `["spec/pagination/links-enumerate-collection"]` the pagination walk. It + * applies after `tiers` and `only`, so it removes rules from what they + * select, and a rule both name is left out. A rule it leaves out is left out + * of the run and out of the report, as one `only` does not select is. + * + * @since 0.17.0 + */ + readonly skip?: ReadonlyArray | undefined; + /** + * The longest the kit waits for one request to be answered and its body + * read, as any `Duration.Input` (`"5 seconds"`, `Duration.seconds(5)`, a + * number of milliseconds). A request still unanswered then is interrupted — + * over `fetch`, aborted — and reads as a lost response: no status, with a + * transport error naming the timeout, which every rule treats as it treats + * any transport failure. Each redirect hop the kit follows is a request with + * a timeout of its own; a client's own `retry` runs within one. + * + * Defaults to thirty seconds, for `verify` and `verifyWith` alike. + * `"Infinity"` (or `Duration.infinity`) turns it off: the kit then waits as + * long as the transport does, so a server that holds a response open holds + * the run. A duration that is not positive — zero, negative or `NaN` — is a + * defect: the run dies before it sends a request. + * + * @since 0.17.0 + */ + readonly requestTimeout?: Duration.Input | undefined; /** A name for the report's header. Defaults to the `HttpApi`'s identifier. */ readonly target?: string | undefined; } const DEFAULT_ABSENT_ID = "conformance-absent-id-0000"; -const resolveOptions = (baseUrl: string, options: Options | undefined): Core.ResolvedOptions => ({ +const resolveOptions = ( + baseUrl: string, + options: Options, + requestTimeout: Option.Option, +): Core.ResolvedOptions => ({ baseUrl, - ids: options?.ids ?? {}, - paths: options?.paths ?? {}, - absentId: options?.absentId ?? DEFAULT_ABSENT_ID, - extensions: options?.extensions ?? [], - mutating: options?.mutating ?? false, - writes: options?.writes ?? {}, + ids: options.ids ?? {}, + paths: options.paths ?? {}, + absentId: options.absentId ?? DEFAULT_ABSENT_ID, + extensions: options.extensions ?? [], + mutating: options.mutating ?? false, + writes: options.writes ?? {}, + requestTimeout, }); const selected = (options: Options | undefined): ReadonlyArray => { const tiers = options?.tiers; const only = options?.only; + const skip = options?.skip ?? []; + const matches = (prefixes: ReadonlyArray, id: string): boolean => + prefixes.some((prefix) => id.startsWith(prefix)); return implementations.filter((implementation) => { - if (tiers !== undefined && !tiers.includes(implementation.rule.tier)) return false; - if (only !== undefined && !only.some((prefix) => implementation.rule.id.startsWith(prefix))) - return false; - return true; + const { id, tier } = implementation.rule; + if (tiers !== undefined && !tiers.includes(tier)) return false; + if (only !== undefined && !matches(only, id)) return false; + return !matches(skip, id); }); }; @@ -801,12 +841,19 @@ const execute = ( options: Options | undefined, ): Effect.Effect => Effect.gen(function* () { - const resolved = resolveOptions(baseUrl, options); + const resolved = resolveOptions( + baseUrl, + options ?? {}, + yield* Core.requestTimeoutOf(Option.fromUndefinedOr(options?.requestTimeout)), + ); const observed = MutableHashSet.empty(); const ctx: Core.Context = { endpoints: Core.discover(api), options: resolved, - request: Core.observing(Core.runner(client, resolved.baseUrl), observed), + request: Core.observing( + Core.runner(client, resolved.baseUrl, resolved.requestTimeout), + observed, + ), writes: [], observed, }; @@ -978,8 +1025,9 @@ export const verify = Effect.fnUntraced(function* < * runs each request through it as its `execute` does, so its `retry` still * re-prepares a request, and a `Location` on the origin it sends to is * followed as one on the base URL's. - * There is no per-request timeout: a server that holds a response open holds - * the run. + * Each request has {@link Options.requestTimeout} (thirty seconds unless the + * run says otherwise) to be answered; a request it cuts short reads as a lost + * response. * * @example * ```ts @@ -1000,7 +1048,11 @@ export const verify = Effect.fnUntraced(function* < * return yield* Conformance.verifyWith(api, { * client, * baseUrl: "https://api.example.com", - * ids: { articles: "1" } + * ids: { articles: "1" }, + * // a live server: give up on a request after ten seconds, not thirty… + * requestTimeout: "10 seconds", + * // …and leave out the rules that walk its collections and crawl its links + * skip: ["spec/pagination/links-enumerate-collection", "spec/links/"] * }) * }).pipe(Effect.provide(FetchHttpClient.layer)) * diff --git a/src/internal/conformance/core.test.ts b/src/internal/conformance/core.test.ts index ce5fa09c..4f248d9f 100644 --- a/src/internal/conformance/core.test.ts +++ b/src/internal/conformance/core.test.ts @@ -1,19 +1,25 @@ import { describe, expect, it } from "vite-plus/test"; import * as Cause from "effect/Cause"; +import * as Duration from "effect/Duration"; import * as Effect from "effect/Effect"; import * as Exit from "effect/Exit"; +import * as Fiber from "effect/Fiber"; import * as MutableHashSet from "effect/MutableHashSet"; import * as Option from "effect/Option"; import { HttpClient, HttpClientResponse, HttpServerError, HttpServerRequest } from "effect/http"; +import { TestClock } from "effect/testing"; import { answeredAsDeployed, type Context, + DEFAULT_REQUEST_TIMEOUT, finding, type Observation, pass, type Rule, + requestTimeoutOf, runner, skip, + timedOutNote, undecidedByRedirects, } from "./core.js"; import { budgeted } from "./crawl.js"; @@ -91,7 +97,7 @@ const served = (answer: (path: string) => readonly [status: number, location?: s ); }), ); - return { sent, request: runner(client, "http://origin.test") }; + return { sent, request: runner(client, "http://origin.test", Option.none()) }; }; /** `/n` redirects (`status`) to `/n+1` until `/end`, which answers 200. */ @@ -180,6 +186,232 @@ describe("runner (redirects)", () => { }); }); +/** + * What a {@link slow} client answers a path with: a status (and `Location`) + * after `after`, or never; `held` sends the headers and then holds the body + * open. + */ +interface Slow { + readonly after: Duration.Input | "never"; + readonly status?: number; + readonly location?: string; + readonly held?: boolean; +} + +/** + * A runner with a one-second `requestTimeout` over a client that answers each + * path as `answer` says. `sent` is every path requested, `interrupted` how + * many requests were cut short. + */ +const slow = (answer: (path: string) => Slow, timeout = Option.some(Duration.seconds(1))) => { + const sent: Array = []; + let interrupted = 0; + const client = HttpClient.make((request, url) => { + sent.push(url.pathname); + const { after, status = 200, location, held = false } = answer(url.pathname); + const respond = Effect.sync(() => + HttpClientResponse.fromWeb( + request, + new Response(held ? new ReadableStream() : status === 200 ? '{"data":null}' : null, { + status, + headers: location === undefined ? {} : { location }, + }), + ), + ); + return (after === "never" ? Effect.never : Effect.delay(respond, after)).pipe( + Effect.onInterrupt(() => + Effect.sync(() => { + interrupted += 1; + }), + ), + ); + }); + return { + sent, + interrupted: () => interrupted, + request: runner(client, "http://origin.test", timeout), + }; +}; + +/** + * Runs `probe` under the test clock, advancing it by each of `steps` in turn; + * `settled` says, after each step, whether the probe had its observation yet. + */ +const clocked = (probe: Effect.Effect, steps: ReadonlyArray) => + valueOf( + Effect.gen(function* () { + const fiber = yield* Effect.forkChild(probe); + const settled = yield* Effect.forEach( + steps, + (step) => Effect.map(TestClock.adjust(step), () => fiber.pollUnsafe() !== undefined), + { concurrency: 1 }, + ); + return { settled, observation: yield* Fiber.join(fiber) }; + }).pipe(Effect.provide(TestClock.layer())), + ); + +describe("runner (requestTimeout)", () => { + it("reads a request still unanswered at the timeout as a lost response naming it", async () => { + const target = slow(() => ({ after: "never" })); + const { settled, observation } = await clocked(target.request({ url: "/0" }), [ + "999 millis", + "1 millis", + ]); + expect(settled).toEqual([false, true]); + expect(observation.status).toBe(0); + expect(observation.transportError).toBe(timedOutNote(Duration.seconds(1))); + expect(observation.transportError).toBe("no answer within 1s, the run's `requestTimeout`"); + // A transport error, not a redirect the kit did not follow. + expect(observation.unfollowed).toEqual(Option.none()); + expect(target.interrupted()).toBe(1); + }); + + it("reads an answer within the timeout as before", async () => { + const target = slow(() => ({ after: "999 millis" })); + const { observation } = await clocked(target.request({ url: "/0" }), ["999 millis"]); + expect(observation.status).toBe(200); + expect(observation.transportError).toBeUndefined(); + expect(target.interrupted()).toBe(0); + }); + + it("cuts short an answer whose body is held open", async () => { + const target = slow(() => ({ after: 0, held: true })); + const { settled, observation } = await clocked(target.request({ url: "/0" }), [ + "999 millis", + "1 millis", + ]); + expect(settled).toEqual([false, true]); + expect(observation.status).toBe(0); + expect(observation.transportError).toBe(timedOutNote(Duration.seconds(1))); + }); + + it("gives each redirect hop a timeout of its own, and names the hop it cut short", async () => { + // Each hop takes 600ms, so the chain takes 1.2s, past one timeout but + // within each hop's. + const within = slow((path) => + path === "/0" + ? { after: "600 millis", status: 308, location: "/1" } + : { after: "600 millis" }, + ); + const answered = await clocked(within.request({ url: "/0" }), ["600 millis", "600 millis"]); + expect(answered.observation.status).toBe(200); + expect(answered.observation.redirects).toEqual(["http://origin.test/1"]); + // A hop that never answers is cut short a second after it is requested, + // and counts as a request (in `redirects`), as any hop the kit sent does. + const held = slow((path) => + path === "/0" ? { after: "600 millis", status: 308, location: "/1" } : { after: "never" }, + ); + const lost = await clocked(held.request({ url: "/0" }), [ + "600 millis", + "999 millis", + "1 millis", + ]); + expect(lost.settled).toEqual([false, false, true]); + expect(lost.observation.status).toBe(0); + expect(lost.observation.redirects).toEqual(["http://origin.test/1"]); + expect(lost.observation.transportError).toBe(timedOutNote(Duration.seconds(1))); + expect(lost.observation.unfollowed).toEqual(Option.none()); + expect(held.sent).toEqual(["/0", "/1"]); + }); + + it("waits thirty seconds by default", async () => { + const target = slow(() => ({ after: "never" }), await valueOf(requestTimeoutOf(Option.none()))); + const { settled, observation } = await clocked(target.request({ url: "/0" }), [ + "29999 millis", + "1 millis", + ]); + expect(settled).toEqual([false, true]); + expect(observation.status).toBe(0); + expect(observation.transportError).toBe("no answer within 30s, the run's `requestTimeout`"); + expect(target.interrupted()).toBe(1); + }); + + it("runs a client's own `retry` within the timeout, not each attempt under one", async () => { + // The first attempt never answers and the second would at once: were the + // timeout inside the retry, failing the first attempt, the second would + // answer 200. It bounds the client's whole `execute`, so the run reads a + // lost response after one attempt, interrupted. + let attempts = 0; + let interrupted = 0; + const client = HttpClient.retry( + HttpClient.make((request) => { + attempts += 1; + return attempts === 1 + ? Effect.never.pipe( + Effect.onInterrupt(() => + Effect.sync(() => { + interrupted += 1; + }), + ), + ) + : Effect.succeed(HttpClientResponse.fromWeb(request, new Response('{"data":null}'))); + }), + { times: 1 }, + ); + const request = runner(client, "http://origin.test", Option.some(Duration.seconds(1))); + const { settled, observation } = await clocked(request({ url: "/0" }), [ + "999 millis", + "1 millis", + ]); + expect(settled).toEqual([false, true]); + expect(observation.status).toBe(0); + expect(observation.transportError).toBe(timedOutNote(Duration.seconds(1))); + expect(attempts).toBe(1); + expect(interrupted).toBe(1); + }); + + it("waits as long as the transport does when the run turns it off", async () => { + const target = slow( + () => ({ after: "1 hour" }), + await valueOf(requestTimeoutOf(Option.some("Infinity"))), + ); + const { settled, observation } = await clocked(target.request({ url: "/0" }), [ + "59 minutes", + "1 minute", + ]); + expect(settled).toEqual([false, true]); + expect(observation.status).toBe(200); + }); +}); + +describe("requestTimeoutOf", () => { + it("defaults to thirty seconds and keeps a positive duration", async () => { + expect(await valueOf(requestTimeoutOf(Option.none()))).toEqual( + Option.some(Duration.seconds(30)), + ); + expect(DEFAULT_REQUEST_TIMEOUT).toEqual(Duration.seconds(30)); + expect(await valueOf(requestTimeoutOf(Option.some("5 seconds")))).toEqual( + Option.some(Duration.seconds(5)), + ); + expect(await valueOf(requestTimeoutOf(Option.some(1)))).toEqual( + Option.some(Duration.millis(1)), + ); + }); + + it.each(["Infinity", Duration.infinity] as const)("reads %s as no timeout", async (off) => { + expect(await valueOf(requestTimeoutOf(Option.some(off)))).toEqual(Option.none()); + }); + + it.each([ + [0, "0"], + [-1, "-1"], + [Number.NaN, "NaN"], + ["-5 seconds", '"-5 seconds"'], + ["-Infinity", '"-Infinity"'], + [Duration.zero, null], + ] as const)("dies on a duration that is not positive: %s", async (input, named) => { + const exit = await valueOf(Effect.exit(requestTimeoutOf(Option.some(input)))); + const cause = Exit.isFailure(exit) ? Option.some(exit.cause) : Option.none(); + expect(Option.map(cause, Cause.hasDies)).toEqual(Option.some(true)); + const message = Option.match(cause, { + onNone: () => "", + onSome: (c) => String(Cause.squash(c)), + }); + expect(message).toContain("`requestTimeout` must be a positive duration"); + expect(named === null || message.endsWith(`got ${named}`)).toBe(true); + }); +}); + describe("budgeted (each redirect is a request)", () => { const io = (budget: number, answer: (path: string) => readonly [number, string?]) => { const target = served(answer); @@ -193,6 +425,7 @@ describe("budgeted (each redirect is a request)", () => { extensions: [], mutating: false, writes: {}, + requestTimeout: Option.none(), }, writes: [], observed: MutableHashSet.empty(), diff --git a/src/internal/conformance/core.ts b/src/internal/conformance/core.ts index 985f0143..34bea953 100644 --- a/src/internal/conformance/core.ts +++ b/src/internal/conformance/core.ts @@ -10,6 +10,7 @@ import { Array as Arr, Cause, + Duration, Effect, Formatter, MutableHashSet, @@ -430,6 +431,11 @@ export interface ResolvedOptions { readonly extensions: ReadonlyArray; readonly mutating: boolean; readonly writes: Readonly>; + /** + * The longest the kit waits for one request's answer, resolved by + * {@link requestTimeoutOf}; none when the run turned it off. + */ + readonly requestTimeout: Option.Option; } // --------------------------------------------------------------------------- @@ -749,6 +755,82 @@ const hopOf = ( }); }; +/** What one request came to: the probe's observation, or a redirect hop to take. */ +type Step = + | { readonly _tag: "Answered"; readonly observation: Observation } + | { readonly _tag: "Hop"; readonly next: { readonly method: string; readonly url: string } }; + +const answered = (observation: Observation): Step => ({ _tag: "Answered", observation }); + +/** + * The note a request `requestTimeout` cut short is named by, in + * `transportError`. + * + * @internal + */ +export const timedOutNote = (timeout: Duration.Duration): string => + `no answer within ${Duration.format(timeout)}, the run's \`requestTimeout\``; + +/** + * How long the kit waits for one request when the run does not say: + * thirty seconds (owner decision on #302). + * + * @internal + */ +export const DEFAULT_REQUEST_TIMEOUT: Duration.Duration = Duration.seconds(30); + +/** + * `Options.requestTimeout` resolved: {@link DEFAULT_REQUEST_TIMEOUT} when + * unset, none (no timeout) for `"Infinity"` or `Duration.infinity`, and the + * duration given otherwise. A duration that is not positive — zero, negative, + * `NaN` (which `Duration` reads as zero) — or that is not a duration at all + * would cut every request short before it was sent, so it is a defect in the + * caller's options, not a run to report. + * + * @internal + */ +export const requestTimeoutOf = ( + given: Option.Option, +): Effect.Effect> => { + if (Option.isNone(given)) return Effect.succeed(Option.some(DEFAULT_REQUEST_TIMEOUT)); + const input = given.value; + const timeout = Duration.fromInput(input); + if (Option.isSome(timeout) && Duration.isPositive(timeout.value)) { + return Effect.succeed(Duration.isFinite(timeout.value) ? timeout : Option.none()); + } + return Effect.die( + new RangeError( + `Conformance: \`requestTimeout\` must be a positive duration, or "Infinity" for no timeout; got ${Formatter.format(input)}`, + ), + ); +}; + +/** + * `request`, one request and the reading of its answer, bounded by `timeout` + * when there is one. A request still unanswered then is interrupted — over + * `fetch`, aborted — and read as a lost response: `status: 0`, with + * {@link timedOutNote} in `transportError` and no redirect in `unfollowed`, so + * a rule reads it as it reads any transport error (not as a redirect the kit + * did not follow). It covers the whole of the client's `execute`, so a + * client's own `retry` runs within it, and the answer's body, so a server + * that sends its headers and holds the body open is cut short too. Each + * redirect hop is a request of its own, with a timeout of its own. + */ +const timed = ( + request: Effect.Effect, + probe: Probe, + redirects: ReadonlyArray, + timeout: Option.Option, +): Effect.Effect => + Option.match(timeout, { + onNone: () => request, + onSome: (bound) => + Effect.map( + Effect.timeoutOption(request, bound), + Option.getOrElse(() => answered(unread(probe, timedOutNote(bound), redirects))), + ), + }); + /** * Builds the probe runner over an `HttpClient`. A transport failure becomes an * `Observation` with `status: 0` rather than an error, so one unreachable @@ -780,10 +862,21 @@ const hopOf = ( * (see {@link transportFollowed}). A failure of `preprocess` itself, with no * request sent, is read as any other transport error. * + * With a `requestTimeout` (none when the run turned it off), each request — + * the first and each hop — has that long to be answered and read, the + * client's `execute` and the body both, and one that is not reads as a lost + * response naming the timeout (see {@link timed}). + * The timeout wraps the client's `postprocess` from outside, so the request + * still runs through the client exactly as its `execute` runs it. + * * @internal */ export const runner = - (client: HttpClient.HttpClient, baseUrl: string): Context["request"] => + ( + client: HttpClient.HttpClient, + baseUrl: string, + requestTimeout: Option.Option, + ): Context["request"] => (probe) => { const method = probe.method ?? "GET"; const headers = probe.headers ?? jsonApiAccept; @@ -793,11 +886,15 @@ export const runner = const record: Probe = { method, url: probe.url, headers, body: probe.body }; const origin = new URL(URL.canParse(baseUrl) ? baseUrl : first).origin; const allowed = probe.requests ?? HOP_CAP + 1; - /** Sends one request, then the hop its redirect leads to, if the kit takes it. */ - const send = ( + /** + * Sends one request and reads its answer: the observation, or the hop its + * redirect leads to when the kit takes it. One request, so what + * `requestTimeout` bounds (see {@link timed}). + */ + const step = ( sent: { readonly method: string; readonly url: string }, redirects: ReadonlyArray, - ): Effect.Effect => + ): Effect.Effect => Effect.gen(function* () { const request = clientRequest(sent.url, { ...record, method: sent.method }); // The request as the client sends it: what its own `preprocess` — where @@ -825,7 +922,7 @@ export const runner = ); // A `preprocess` that failed sent nothing, so no redirect was followed. if (Option.isNone(prepared) && Result.isFailure(result)) { - return unread(record, String(result.failure), redirects); + return answered(unread(record, String(result.failure), redirects)); } const sentAs = Option.getOrElse(prepared, () => request); // Where the transport ended: the request behind its failure or its @@ -846,27 +943,42 @@ export const runner = const sentTo = urlOf(sentAs); const followed = transportFollowed({ method: sentAs.method, url: sentTo }, ended); if (Option.isSome(followed)) { - return unread(record, unfollowedNote(followed.value), redirects, followed); + return answered(unread(record, unfollowedNote(followed.value), redirects, followed)); + } + if (Result.isFailure(result)) { + return answered(unread(record, String(result.failure), redirects)); } - if (Result.isFailure(result)) return unread(record, String(result.failure), redirects); const response = result.success; const location = response.headers["location"]; // An empty `Location` names nothing to follow, as a missing one does. if (!REDIRECTS.includes(response.status) || location === undefined || location === "") { - return yield* answerOf(record, response, redirects); + return answered(yield* answerOf(record, response, redirects)); } const hop = hopOf({ status: response.status, location }, { ...sent, sentTo }, origin, { hops: redirects.length, allowed, }); - return yield* Result.match(hop, { - onFailure: (unfollowed) => - Effect.succeed( + return Result.match(hop, { + onFailure: (unfollowed): Step => + answered( unread(record, unfollowedNote(unfollowed), redirects, Option.some(unfollowed)), ), - onSuccess: (next) => send(next, [...redirects, next.url]), + onSuccess: (next): Step => ({ _tag: "Hop", next }), }); }); + /** + * Sends one request, then the hop its redirect leads to, if the kit takes + * it, each under its own `requestTimeout`. + */ + const send = ( + sent: { readonly method: string; readonly url: string }, + redirects: ReadonlyArray, + ): Effect.Effect => + Effect.flatMap(timed(step(sent, redirects), record, redirects, requestTimeout), (taken) => + Predicate.isTagged(taken, "Answered") + ? Effect.succeed(taken.observation) + : send(taken.next, [...redirects, taken.next.url]), + ); return send({ method, url: first }, []).pipe(Effect.scoped, Effect.orDie); }; diff --git a/src/internal/conformance/write.test.ts b/src/internal/conformance/write.test.ts index e2ace371..d579f31a 100644 --- a/src/internal/conformance/write.test.ts +++ b/src/internal/conformance/write.test.ts @@ -58,6 +58,7 @@ const context = (): Core.Context => { cleanup: (id) => Effect.sync(() => MutableHashMap.remove(store, id)), }, }, + requestTimeout: Option.none(), }, writes: [], observed, From 9bb1a9a4a1a1cb963f7570dd3d6068472ae1bcbd Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 15 Oct 2026 07:04:48 +0000 Subject: [PATCH 2/2] test(Conformance): pin the in-process timeout default; name unreadable durations (#302) A run of `verify` against handlers that never answer is still waiting a millisecond before thirty seconds and cuts each request short at thirty, so the default is pinned for `verify` as well as `verifyWith`; `verify`'s JSDoc says so, as `verifyWith`'s does. A `requestTimeout` that `Duration` cannot read (`"1e3 seconds"`, `".5 seconds"`), which the type admits, dies as a non-positive one does; the JSDoc, README, design note and changeset now name it, the `requestTimeoutOf` table and the run-level test cover it, and the message asks for "a positive duration `Duration` can read". The run-level "dies before it sends a request" tests now use a client that fails at once, so a run that read its options after sending would finish and fail on `sent`, not hang into the test timeout. Part of #302 Claude-Session: https://claude.ai/code/session_0187g2QpzaHqEzX8GJxiJBVc Co-Authored-By: Claude --- .changeset/conformance-skip-and-timeout.md | 2 +- README.md | 2 +- docs/conformance.md | 4 +- src/Conformance.test.ts | 82 ++++++++++++++++++++-- src/Conformance.ts | 8 ++- src/internal/conformance/core.test.ts | 28 +++++--- src/internal/conformance/core.ts | 2 +- 7 files changed, 106 insertions(+), 22 deletions(-) diff --git a/.changeset/conformance-skip-and-timeout.md b/.changeset/conformance-skip-and-timeout.md index fc8dbc2f..65a8535f 100644 --- a/.changeset/conformance-skip-and-timeout.md +++ b/.changeset/conformance-skip-and-timeout.md @@ -2,4 +2,4 @@ "@thomasfosterau/effect-jsonapi": minor --- -**`Conformance.Options` gains `skip` and `requestTimeout`, two ways to bound a run against a live server (#302).** `skip` is a list of rule-id prefixes to leave out, the complement of `only` and matched the same way (a plain string prefix of the id): `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl. It removes rules from what `tiers` and `only` select, so a rule both `only` and `skip` name is left out, and a rule it leaves out is absent from the report, as one `only` does not select is. `requestTimeout` takes any positive `Duration.Input` and bounds each request the kit sends — the client's whole `execute`, a `retry` it wraps itself in included, and the reading of the body — with each redirect hop a request of its own. A request still unanswered then is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout, which every rule treats as any other transport error and never as a redirect the kit did not follow. **This changes the default for every run:** `requestTimeout` defaults to thirty seconds, for `verify` and `verifyWith` alike, so a run that sets nothing now reads a request unanswered after thirty seconds as a lost response where it used to wait for as long as the transport did. `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off and restores the old behaviour; a duration that is not positive (zero, negative, `NaN`) is a defect. `skip` is unset by default, leaving out nothing. +**`Conformance.Options` gains `skip` and `requestTimeout`, two ways to bound a run against a live server (#302).** `skip` is a list of rule-id prefixes to leave out, the complement of `only` and matched the same way (a plain string prefix of the id): `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl. It removes rules from what `tiers` and `only` select, so a rule both `only` and `skip` name is left out, and a rule it leaves out is absent from the report, as one `only` does not select is. `requestTimeout` takes any positive `Duration.Input` and bounds each request the kit sends — the client's whole `execute`, a `retry` it wraps itself in included, and the reading of the body — with each redirect hop a request of its own. A request still unanswered then is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout, which every rule treats as any other transport error and never as a redirect the kit did not follow. **This changes the default for every run:** `requestTimeout` defaults to thirty seconds, for `verify` and `verifyWith` alike, so a run that sets nothing now reads a request unanswered after thirty seconds as a lost response where it used to wait for as long as the transport did. `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off and restores the old behaviour; a duration that is not positive (zero, negative, `NaN`), or one `Duration` cannot read (e.g. `"1e3 seconds"`), is a defect. `skip` is unset by default, leaving out nothing. diff --git a/README.md b/README.md index 0980ea5f..663e2523 100644 --- a/README.md +++ b/README.md @@ -2006,7 +2006,7 @@ Use `Conformance.assert(report)` as a gate: it fails with a `ContractViolation` **Against a live server, bound the run.** `verifyWith` runs the identical rules over any `HttpClient` and `baseUrl`. Two options bound what that costs the server: - `skip` — rule-id prefixes to leave out, the complement of `only` and matched the same way: `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl, the rules that make the most requests. It removes rules from what `tiers` and `only` select, and a rule it leaves out is absent from the report, as one `only` does not select is. -- `requestTimeout` — a `Duration.Input` (`"10 seconds"`) bounding each request, its body included; each redirect hop gets its own. A request still unanswered then is aborted and reads as a lost response naming the timeout, which every rule treats as any transport error. Thirty seconds by default, for `verify` and `verifyWith` alike; `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off, so the kit waits as long as the transport does. A duration that is not positive is a defect. +- `requestTimeout` — a `Duration.Input` (`"10 seconds"`) bounding each request, its body included; each redirect hop gets its own. A request still unanswered then is aborted and reads as a lost response naming the timeout, which every rule treats as any transport error. Thirty seconds by default, for `verify` and `verifyWith` alike; `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off, so the kit waits as long as the transport does. A duration that is not positive, or one `Duration` cannot read (e.g. `"1e3 seconds"`), is a defect. Full design note, including the coverage ledger and what it would take to close each gap: [`docs/conformance.md`](./docs/conformance.md). diff --git a/docs/conformance.md b/docs/conformance.md index 6e59fef1..6048a5b9 100644 --- a/docs/conformance.md +++ b/docs/conformance.md @@ -129,7 +129,7 @@ Conformance.verifyWith(api, { **`requestTimeout` bounds each request in time.** A request still unanswered when it runs out is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout (``no response (no answer within 30s, the run's `requestTimeout`)``). Every rule treats it exactly as it treats a refused connection or a reset — `spec/target/reachable` reports it, the pagination walk and the link crawl leave what it cut short not decided — and never as a redirect the kit did not follow, so the [#333](https://github.com/thomasfosterau/effect-jsonapi/pull/333#issuecomment-6181006987) rule that a request a redirect left unanswered skips the rule that sent it does not apply. It bounds one request: the client's whole `execute`, a `retry` the client wraps itself in included (the kit still runs each request through the client exactly as its `execute` does, §1.2), and the reading of the answer's body, so a server that sends its headers and holds the body open is cut short too. Each redirect hop the kit follows is a request of its own, with a timeout of its own and counting against the rule's budget as before, so one probe waits at most six timeouts (the first request and five hops). A request it cuts short was still sent, and still counts against the budget. A create it cuts short is taken to have created nothing, as a non-2xx one is (§1.4): the server may have created it all the same. It does not bound `cleanup`, which is not a request the kit sends (§1.4). -**It defaults to thirty seconds**, for `verify` and `verifyWith` alike (owner decision on [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302#issuecomment-6186007738)), so a server that holds a response open costs a run at most that per request, and a run that sets nothing is bounded too. Any positive `Duration.Input` overrides it. `requestTimeout: "Infinity"` (or `Duration.infinity`, the same value) turns it off: the kit then waits as long as the transport does, as it did before the option existed, so a server that holds a response open holds the run. `Infinity` is the spelling because it is already a `Duration.Input`, so the option keeps one type and needs no second sentinel. A duration that is not positive — zero, a negative one, or `NaN`, which `Duration` reads as zero — would cut every request short before it was answered, so it is a defect in the options: the run dies before it sends a request, rather than reporting every rule undecided. +**It defaults to thirty seconds**, for `verify` and `verifyWith` alike (owner decision on [#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302#issuecomment-6186007738)), so a server that holds a response open costs a run at most that per request, and a run that sets nothing is bounded too. Any positive `Duration.Input` overrides it. `requestTimeout: "Infinity"` (or `Duration.infinity`, the same value) turns it off: the kit then waits as long as the transport does, as it did before the option existed, so a server that holds a response open holds the run. `"Infinity"` is the spelling because it is already a `Duration.Input`, so the option keeps one type and needs no second sentinel. A duration that is not positive — zero, a negative one, or `NaN`, which `Duration` reads as zero — would cut every request short before it was answered, so it is a defect in the options, as is one `Duration` cannot read (e.g. `"1e3 seconds"`): the run dies before it sends a request, rather than reporting every rule undecided. ## 2. Two tiers, separately runnable @@ -351,7 +351,7 @@ A kit that fails everything catches every bug and is worthless. The control — **The link rules get their own server** (`linked` in `Conformance.test.ts`), because the main fixture's resources carry no links: two articles, each with a `self`, a to-one `author`, a to-many `tags` and a links-only `comments`, every link resolving to what it names. Against it all three rules pass, with the requests asserted, so a pass is shown to have followed each link once (a link to an endpoint already fetched costs nothing; `related-matches-linkage` does not follow the links-only `comments`). Then each way a link breaks, one at a time, with the finding pinned: a `self` that leads to another resource, to a 404 or to `data: null`; a `related` that answers something other than resource data (`related-dereferences`); and a `related` that omits a resource its linkage names, answers one it does not name, answers another to-one resource, or answers a resource where the linkage is `null` (`related-matches-linkage`, while `related-dereferences` passes the same answer). A `related` that answers 404 or is not a link at all fails `related-dereferences` once and is left not decided by `related-matches-linkage`, which skips when no relationship carries linkage. The subset relaxations are pinned both ways — a page of the related collection, and a page of linkage, each passing when it is consistent and failing when it is not — and two pages are not compared; nor are an answer, a resource or linkage holding an item without a string `type` and `id`. Link objects, a relative link (one that resolves differently against the document and against the base URL) and an absolute link to the run's own origin are followed; a link that is `{"toString":1}` is a finding; a link to another origin is never requested (every request is asserted to go to the run's origin) and is noted, or skipped when it is the only one; a `javascript:` link is named by its scheme. A `fetch` that would follow a redirect off the origin is never let to, even when the run's own `RequestInit` says `redirect: "follow"`, and the rule is skipped, naming the `Location`, never passed on the links that did decide; an `http` → `https` redirect on the same host is treated the same way, and so is a read endpoint redirected off the origin while every link it did find decides. A redirect within the origin is followed, its final answer graded (a different resource there is a finding) and its hop counted; a chain of five hops is followed and one of six skips the rule; a relative link in a redirected answer resolves against where the answer came from; and an answer a transport fetched from another origin all the same is not read, and skips the rule, naming where the transport ended up, rather than passing it on the links that did decide. A `related` answering `{"data":["x"]}`, `{"data":"x"}` or `{"data":[{"id":"1"}]}` is a finding; `{"data":null}` passes. A label the linkage names is quoted cut like any other. A 503, a 500, a 429, a 408, a lost response and a 403 leave the link not decided. The budget is pinned at exactly 200 requests over 250 articles (skipped, naming it), and over 201 read endpoints, one never searched (skipped, naming it, and not counting a read endpoint whose URL does not resolve as searched); a finding made before it ran out still fails the rule. Twenty-five broken links list ten findings and count the rest, and a 300-character id is quoted cut. The roles searched are pinned: a related endpoint's answer is, a relationship endpoint is never requested. The skips name what they searched. An in-process case holds the real router to the rule, including a link to a path no route serves, which fails it rather than killing it; `src/internal/conformance/core.test.ts` pins that only `RouteNotFound` is answered so, any other router failure dying. -**`skip` and `requestTimeout` are pinned at both ends** ([#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). `Conformance.test.ts` asserts that `skip` leaves out exactly the rules its prefixes match (the pagination walk, the link crawl and a section among them), adding no `Skipped` outcome; that it removes rules from what `only` and `tiers` select; and that a rule both `only` and `skip` name is left out. Against a canned server that never answers, a run with a 10ms `requestTimeout` finishes, interrupts every request it sent, and reports exactly what a transport that fails at once reports, but for the note naming the timeout; against the conformant server, a timeout no request reaches changes no outcome, through a client's `retry` too. `core.test.ts` pins the runner under the test clock: a request is still pending a millisecond before the timeout and a lost response naming it at the timeout, with no redirect recorded; a body held open is cut short too; each redirect hop has a timeout of its own (two 600ms hops pass under a one-second timeout, and a hop that never answers is cut short a second after it is requested, counting as a request sent); a client's own `retry` runs within the timeout (a first attempt that hangs is cut short with the run's one request, and the second attempt, which would answer, is never made); and with the timeout turned off (`"Infinity"`) the runner waits as long as the transport does. The default is pinned twice: `core.test.ts` holds `requestTimeoutOf` to thirty seconds when unset, no timeout for `"Infinity"` and `Duration.infinity`, and a defect for zero, a negative duration and `NaN`, and the runner to a request still pending a millisecond before thirty seconds; `Conformance.test.ts` holds a run that sets nothing to cutting each request short at thirty seconds under the test clock, a run that turns it off to still waiting an hour later, and a run with a non-positive timeout to dying before it sends a request. +**`skip` and `requestTimeout` are pinned at both ends** ([#302](https://github.com/thomasfosterau/effect-jsonapi/issues/302)). `Conformance.test.ts` asserts that `skip` leaves out exactly the rules its prefixes match (the pagination walk, the link crawl and a section among them), adding no `Skipped` outcome; that it removes rules from what `only` and `tiers` select; and that a rule both `only` and `skip` name is left out. Against a canned server that never answers, a run with a 10ms `requestTimeout` finishes, interrupts every request it sent, and reports exactly what a transport that fails at once reports, but for the note naming the timeout; against the conformant server, a timeout no request reaches changes no outcome, through a client's `retry` too. `core.test.ts` pins the runner under the test clock: a request is still pending a millisecond before the timeout and a lost response naming it at the timeout, with no redirect recorded; a body held open is cut short too; each redirect hop has a timeout of its own (two 600ms hops pass under a one-second timeout, and a hop that never answers is cut short a second after it is requested, counting as a request sent); a client's own `retry` runs within the timeout (a first attempt that hangs is cut short with the run's one request, and the second attempt, which would answer, is never made); and with the timeout turned off (`"Infinity"`) the runner waits as long as the transport does. The default is pinned twice: `core.test.ts` holds `requestTimeoutOf` to thirty seconds when unset, no timeout for `"Infinity"` and `Duration.infinity`, and a defect for zero, a negative duration, `NaN` and a duration `Duration` cannot read (`"1e3 seconds"`), and the runner to a request still pending a millisecond before thirty seconds; `Conformance.test.ts` holds a run that sets nothing to cutting each request short at thirty seconds under the test clock, through `verifyWith` and in process through `verify` alike (handlers that never answer, still waiting a millisecond before), a run that turns it off to still waiting an hour later, and a run with a non-positive or unreadable timeout to dying before it sends a request, against a client that answers at once, so a run that sent one first would finish and be caught rather than hang. ## 10. Open questions diff --git a/src/Conformance.test.ts b/src/Conformance.test.ts index 5cefd9fb..047e5518 100644 --- a/src/Conformance.test.ts +++ b/src/Conformance.test.ts @@ -33,6 +33,7 @@ import * as Group from "./Group.js"; import * as Handlers from "./Handlers.js"; import * as Middleware from "./Middleware.js"; import * as Query from "./Query.js"; +import * as Registry from "./Registry.js"; import * as Relationship from "./Relationship.js"; import * as Resource from "./Resource.js"; import { ATOMIC_EXTENSION_URI, MEDIA_TYPE } from "./internal/media.js"; @@ -886,6 +887,70 @@ describe("Conformance: requestTimeout", () => { ); }); + it("waits thirty seconds for each request in process too", async () => { + // Handlers that never answer, through the real router: the default is + // the same for `verify` as for `verifyWith`. + let interrupted = 0; + const hang = Effect.never.pipe( + Effect.onInterrupt(() => + Effect.sync(() => { + interrupted += 1; + }), + ), + ); + const Live = HttpApiBuilder.group(api, "articles", (handlers) => + handlers.handle("get", () => hang).handle("list", () => hang), + ).pipe(Layer.provideMerge(Middleware.layer)); + const exit = await Effect.runPromiseExit( + Effect.gen(function* () { + const fiber = yield* Effect.forkChild( + Conformance.verify(api, { ids: { articles: "1" }, only: ["spec/target/reachable"] }), + ); + // After each step: whether the run had finished, and how many + // requests had been cut short. + const steps = yield* Effect.forEach( + ["29999 millis", "1 millis", "30 seconds"] as const, + (step) => + Effect.map(TestClock.adjust(step), () => ({ + settled: fiber.pollUnsafe() !== undefined, + interrupted, + })), + { concurrency: 1 }, + ); + const report = + steps.at(-1)?.settled === true ? Option.some(yield* Fiber.join(fiber)) : Option.none(); + yield* Fiber.interrupt(fiber); + return { steps, report }; + }).pipe( + Effect.provide(Layer.mergeAll(Live, Registry.platformLayer, TestClock.layer())), + Effect.scoped, + ), + ); + expect(Exit.isSuccess(exit)).toBe(true); + if (Exit.isFailure(exit)) return; + // Still waiting a millisecond before thirty seconds; the first request is + // cut short at thirty, the second thirty after. + expect(exit.value.steps).toEqual([ + { settled: false, interrupted: 0 }, + { settled: false, interrupted: 1 }, + { settled: true, interrupted: 2 }, + ]); + const note = "no answer within 30s, the run's `requestTimeout`"; + expect( + Option.map(exit.value.report, (report) => outcomeFor(report, "spec/target/reachable")), + ).toEqual( + Option.some( + expect.objectContaining({ + _tag: "Failed", + findings: [ + expect.objectContaining({ observed: `no response (${note})` }), + expect.objectContaining({ observed: `no response (${note})` }), + ], + }), + ), + ); + }); + it.each(["Infinity", Duration.infinity] as const)( "waits as long as the transport does when the run turns it off with %s", async (requestTimeout) => { @@ -899,19 +964,28 @@ describe("Conformance: requestTimeout", () => { }, ); - it.each([0, -1, Number.NaN, "-Infinity"] as const)( + it.each([0, -1, Number.NaN, "-Infinity", "1e3 seconds"] as const)( "dies on a `requestTimeout` of %s before it sends a request", async (requestTimeout) => { let sent = 0; + // The client answers every request at once, so a run that sent one + // before reading its options still finishes, and fails on `sent` + // rather than hanging. const exit = await Effect.runPromiseExit( Conformance.verifyWith(api, { - client: HttpClient.make(() => - Effect.sync(() => { + client: HttpClient.make((request) => + Effect.suspend(() => { sent += 1; - }).pipe(Effect.andThen(Effect.never)), + return Effect.fail( + new HttpClientError.HttpClientError({ + reason: new HttpClientError.TransportError({ request, description: "reset" }), + }), + ); + }), ), baseUrl: "http://fixture.test", ids: { articles: "1" }, + only: ["spec/target/reachable"], requestTimeout, }), ); diff --git a/src/Conformance.ts b/src/Conformance.ts index 665e05da..7da6c6fc 100644 --- a/src/Conformance.ts +++ b/src/Conformance.ts @@ -786,8 +786,9 @@ export interface Options { * Defaults to thirty seconds, for `verify` and `verifyWith` alike. * `"Infinity"` (or `Duration.infinity`) turns it off: the kit then waits as * long as the transport does, so a server that holds a response open holds - * the run. A duration that is not positive — zero, negative or `NaN` — is a - * defect: the run dies before it sends a request. + * the run. A duration that is not positive — zero, negative or `NaN` — or + * one `Duration` cannot read (e.g. `"1e3 seconds"`) is a defect: the run + * dies before it sends a request. * * @since 0.17.0 */ @@ -899,6 +900,9 @@ const execute = ( * in-memory `HttpClient` over the real router, so requests go through request * decoding, routing, middleware, the handler and response encoding, and the * rules observe raw statuses, media types and bodies. No port is bound. + * Each request has {@link Options.requestTimeout} (thirty seconds unless the + * run says otherwise) to be answered; a request it cuts short reads as a lost + * response. * * Use {@link verifyWith} to run the identical rules against a deployed server. * diff --git a/src/internal/conformance/core.test.ts b/src/internal/conformance/core.test.ts index 4f248d9f..676bad5d 100644 --- a/src/internal/conformance/core.test.ts +++ b/src/internal/conformance/core.test.ts @@ -398,18 +398,24 @@ describe("requestTimeoutOf", () => { [Number.NaN, "NaN"], ["-5 seconds", '"-5 seconds"'], ["-Infinity", '"-Infinity"'], + // The type admits these, but `Duration` cannot read them. + ["1e3 seconds", '"1e3 seconds"'], + [".5 seconds", '".5 seconds"'], [Duration.zero, null], - ] as const)("dies on a duration that is not positive: %s", async (input, named) => { - const exit = await valueOf(Effect.exit(requestTimeoutOf(Option.some(input)))); - const cause = Exit.isFailure(exit) ? Option.some(exit.cause) : Option.none(); - expect(Option.map(cause, Cause.hasDies)).toEqual(Option.some(true)); - const message = Option.match(cause, { - onNone: () => "", - onSome: (c) => String(Cause.squash(c)), - }); - expect(message).toContain("`requestTimeout` must be a positive duration"); - expect(named === null || message.endsWith(`got ${named}`)).toBe(true); - }); + ] as const)( + "dies on a duration that is not positive, or not readable: %s", + async (input, named) => { + const exit = await valueOf(Effect.exit(requestTimeoutOf(Option.some(input)))); + const cause = Exit.isFailure(exit) ? Option.some(exit.cause) : Option.none(); + expect(Option.map(cause, Cause.hasDies)).toEqual(Option.some(true)); + const message = Option.match(cause, { + onNone: () => "", + onSome: (c) => String(Cause.squash(c)), + }); + expect(message).toContain("`requestTimeout` must be a positive duration"); + expect(named === null || message.endsWith(`got ${named}`)).toBe(true); + }, + ); }); describe("budgeted (each redirect is a request)", () => { diff --git a/src/internal/conformance/core.ts b/src/internal/conformance/core.ts index 34bea953..1f37a3a5 100644 --- a/src/internal/conformance/core.ts +++ b/src/internal/conformance/core.ts @@ -800,7 +800,7 @@ export const requestTimeoutOf = ( } return Effect.die( new RangeError( - `Conformance: \`requestTimeout\` must be a positive duration, or "Infinity" for no timeout; got ${Formatter.format(input)}`, + `Conformance: \`requestTimeout\` must be a positive duration \`Duration\` can read, or "Infinity" for no timeout; got ${Formatter.format(input)}`, ), ); };