Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/conformance-skip-and-timeout.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@thomasfosterau/effect-jsonapi": minor
---

**`Conformance.Options` gains `skip` and `requestTimeout`, two ways to bound a run against a live server (#302).** `skip` is a list of rule-id prefixes to leave out, the complement of `only` and matched the same way (a plain string prefix of the id): `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl. It removes rules from what `tiers` and `only` select, so a rule both `only` and `skip` name is left out, and a rule it leaves out is absent from the report, as one `only` does not select is. `requestTimeout` takes any positive `Duration.Input` and bounds each request the kit sends — the client's whole `execute`, a `retry` it wraps itself in included, and the reading of the body — with each redirect hop a request of its own. A request still unanswered then is interrupted (over `fetch`, aborted) and reads as a lost response: status `0`, with a transport error naming the timeout, which every rule treats as any other transport error and never as a redirect the kit did not follow. **This changes the default for every run:** `requestTimeout` defaults to thirty seconds, for `verify` and `verifyWith` alike, so a run that sets nothing now reads a request unanswered after thirty seconds as a lost response where it used to wait for as long as the transport did. `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off and restores the old behaviour; a duration that is not positive (zero, negative, `NaN`), or one `Duration` cannot read (e.g. `"1e3 seconds"`), is a defect. `skip` is unset by default, leaving out nothing.
5 changes: 5 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2003,6 +2003,11 @@ Two tiers, separately runnable. `spec` is JSON:API v1.1 alone; `profile` is this

Use `Conformance.assert(report)` as a gate: it fails with a `ContractViolation` carrying the rule, its requirement, its citation and the request that decided it. A rule that died is not a finding about your server, but the run could not decide it either, so `assert` dies on it — with a defect tagged `RuleDied` naming the rule and the defect, the original as its `cause` — rather than passing. If you gate on `summarise(report).failed` or `findings(report)` instead of `assert`, also check `summarise(report).errored === 0`.

**Against a live server, bound the run.** `verifyWith` runs the identical rules over any `HttpClient` and `baseUrl`. Two options bound what that costs the server:

- `skip` — rule-id prefixes to leave out, the complement of `only` and matched the same way: `skip: ["spec/pagination/links-enumerate-collection", "spec/links/"]` leaves out the pagination walk and the link crawl, the rules that make the most requests. It removes rules from what `tiers` and `only` select, and a rule it leaves out is absent from the report, as one `only` does not select is.
- `requestTimeout` — a `Duration.Input` (`"10 seconds"`) bounding each request, its body included; each redirect hop gets its own. A request still unanswered then is aborted and reads as a lost response naming the timeout, which every rule treats as any transport error. Thirty seconds by default, for `verify` and `verifyWith` alike; `requestTimeout: "Infinity"` (or `Duration.infinity`) turns it off, so the kit waits as long as the transport does. A duration that is not positive, or one `Duration` cannot read (e.g. `"1e3 seconds"`), is a defect.

Full design note, including the coverage ledger and what it would take to close each gap: [`docs/conformance.md`](./docs/conformance.md).

## Examples
Expand Down
32 changes: 28 additions & 4 deletions docs/conformance.md

Large diffs are not rendered by default.

326 changes: 306 additions & 20 deletions src/Conformance.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ import {
Array as Arr,
Cause,
Deferred,
Duration,
Effect,
Exit,
Fiber,
Expand All @@ -24,13 +25,15 @@ import {
import { FetchHttpClient, HttpClient, HttpClientError, HttpClientResponse } from "effect/http";
import { HttpClientRequest } from "effect/http";
import { HttpApi, HttpApiBuilder, HttpApiEndpoint, HttpApiGroup } from "effect/http-api";
import { TestClock } from "effect/testing";
import * as Conformance from "./Conformance.js";
import * as Endpoint from "./Endpoint.js";
import * as Filter from "./Filter.js";
import * as Group from "./Group.js";
import * as Handlers from "./Handlers.js";
import * as Middleware from "./Middleware.js";
import * as Query from "./Query.js";
import * as Registry from "./Registry.js";
import * as Relationship from "./Relationship.js";
import * as Resource from "./Resource.js";
import { ATOMIC_EXTENSION_URI, MEDIA_TYPE } from "./internal/media.js";
Expand Down Expand Up @@ -721,13 +724,286 @@ describe("Conformance: discovery and selection", () => {
if (skipped._tag === "Skipped") expect(skipped.reason).toContain(`ids["articles"]`);
});

const ids = (report: Conformance.Report) => report.outcomes.map((outcome) => outcome.rule.id);

it("leaves out the skipped rule ids, matching on prefix as `only` does", async () => {
const all = ids(await verifyCanned(conformantServe));
const skip = ["spec/pagination/links-enumerate-collection", "spec/links/", "spec/content-neg"];
const report = await verifyCanned(conformantServe, { skip });
const left = all.filter((id) => !skip.some((prefix) => id.startsWith(prefix)));
// Each prefix leaves something out, and nothing else is.
expect(skip.every((prefix) => all.some((id) => id.startsWith(prefix)))).toBe(true);
expect(left.length).toBeLessThan(all.length - 5);
expect(ids(report)).toEqual(left);
// A skipped rule is absent from the report, as one `only` did not select
// is — never a `Skipped` outcome that would read as a rule the run tried.
expect(Conformance.summarise(report).skipped).toBe(
Conformance.summarise(await verifyCanned(conformantServe, { only: left })).skipped,
);
});

it("leaves out the skipped rules from what `only` and `tiers` select", async () => {
const selected = ids(
await verifyCanned(conformantServe, { only: ["spec/content-negotiation"] }),
);
const weight = "spec/content-negotiation/accept-weight";
const report = await verifyCanned(conformantServe, {
only: ["spec/content-negotiation"],
skip: [weight],
});
expect(selected.filter((id) => id.startsWith(weight)).length).toBeGreaterThan(1);
expect(ids(report)).toEqual(selected.filter((id) => !id.startsWith(weight)));
// A rule both name is left out; `skip` naming everything leaves nothing.
const both = await verifyCanned(conformantServe, {
only: ["spec/document/top-level-members"],
skip: ["spec/document/top-level-members"],
});
expect(both.outcomes).toEqual([]);
const profile = await verifyCanned(conformantServe, { tiers: ["profile"], skip: ["spec/"] });
expect(profile.outcomes.length).toBeGreaterThan(0);
const spec = await verifyCanned(conformantServe, { tiers: ["spec"], skip: ["spec/"] });
expect(spec.outcomes).toEqual([]);
});

it("records the transport it drove", async () => {
const report = await verifyCanned(conformantServe);
expect(report.transport).toBe("http");
expect(report.target).toBe("fixture");
});
});

describe("Conformance: requestTimeout", () => {
/** A client that never answers, counting the requests the timeout cut short. */
const silent = () => {
let interrupted = 0;
const client = HttpClient.make(() =>
Effect.never.pipe(
Effect.onInterrupt(() =>
Effect.sync(() => {
interrupted += 1;
}),
),
),
);
return { client: () => client, interrupted: () => interrupted };
};

const rules = ["spec/target/reachable", "spec/content-negotiation/response-media-type"];

it("reads a server that never answers as a lost response, rather than hanging", async () => {
const server = silent();
const report = await verifyCanned(
conformantServe,
{ requestTimeout: 10, only: rules },
server.client,
);
const note = "no answer within 10ms, the run's `requestTimeout`";
expect(outcomeFor(report, "spec/target/reachable")).toMatchObject({
_tag: "Failed",
findings: [
{ detail: "/articles/1 produced no HTTP response", observed: `no response (${note})` },
{ detail: "/articles produced no HTTP response", observed: `no response (${note})` },
],
});
// Every request was cut short, not left running.
expect(server.interrupted()).toBe(Conformance.summarise(report).probes);
// Exactly what a transport that fails at once reports, but for the note: a
// timeout is a transport error, never a redirect the kit did not follow.
const lost = await verifyCanned(conformantServe, { only: rules }, () =>
HttpClient.make((request) =>
Effect.fail(
new HttpClientError.HttpClientError({
reason: new HttpClientError.TransportError({ request, description: "reset" }),
}),
),
),
);
const why = (outcomes: ReadonlyArray<Conformance.Outcome>) =>
Arr.map(outcomes, (outcome) =>
Predicate.isTagged(outcome, "Failed")
? {
...outcome,
findings: Arr.map(outcome.findings, (found) => ({
...found,
observed: found.observed.replace(/^no response \(.*\)$/, "no response (…)"),
})),
}
: outcome,
);
expect(why(report.outcomes)).toEqual(why(lost.outcomes));
});

/**
* A run of `reachable` against a server that never answers, under the test
* clock, advanced by each of `steps` in turn; `settled` says, after each
* step, whether the run had finished. A run still going after the last step
* is interrupted.
*/
const clocked = (options: Conformance.Options, steps: ReadonlyArray<Duration.Input>) => {
const server = silent();
return Effect.runPromiseExit(
Effect.gen(function* () {
const fiber = yield* Effect.forkChild(
Conformance.verifyWith(api, {
client: server.client(),
baseUrl: "http://fixture.test",
ids: { articles: "1" },
only: ["spec/target/reachable"],
...options,
}),
);
const settled = yield* Effect.forEach(
steps,
(step) => Effect.map(TestClock.adjust(step), () => fiber.pollUnsafe() !== undefined),
{ concurrency: 1 },
);
const report =
settled.at(-1) === true ? Option.some(yield* Fiber.join(fiber)) : Option.none();
yield* Fiber.interrupt(fiber);
return { settled, report, interrupted: server.interrupted() };
}).pipe(Effect.provide(TestClock.layer())),
);
};

it("waits thirty seconds for each request by default", async () => {
const exit = await clocked({}, ["29999 millis", "1 millis", "30 seconds"]);
expect(Exit.isSuccess(exit)).toBe(true);
if (Exit.isFailure(exit)) return;
const run = exit.value;
// The first request is cut short at thirty seconds, the second thirty after.
expect(run.settled).toEqual([false, false, true]);
expect(run.interrupted).toBe(2);
const note = "no answer within 30s, the run's `requestTimeout`";
expect(Option.map(run.report, (report) => outcomeFor(report, "spec/target/reachable"))).toEqual(
Option.some(
expect.objectContaining({
_tag: "Failed",
findings: [
expect.objectContaining({ observed: `no response (${note})` }),
expect.objectContaining({ observed: `no response (${note})` }),
],
}),
),
);
});

it("waits thirty seconds for each request in process too", async () => {
// Handlers that never answer, through the real router: the default is
// the same for `verify` as for `verifyWith`.
let interrupted = 0;
const hang = Effect.never.pipe(
Effect.onInterrupt(() =>
Effect.sync(() => {
interrupted += 1;
}),
),
);
const Live = HttpApiBuilder.group(api, "articles", (handlers) =>
handlers.handle("get", () => hang).handle("list", () => hang),
).pipe(Layer.provideMerge(Middleware.layer));
const exit = await Effect.runPromiseExit(
Effect.gen(function* () {
const fiber = yield* Effect.forkChild(
Conformance.verify(api, { ids: { articles: "1" }, only: ["spec/target/reachable"] }),
);
// After each step: whether the run had finished, and how many
// requests had been cut short.
const steps = yield* Effect.forEach(
["29999 millis", "1 millis", "30 seconds"] as const,
(step) =>
Effect.map(TestClock.adjust(step), () => ({
settled: fiber.pollUnsafe() !== undefined,
interrupted,
})),
{ concurrency: 1 },
);
const report =
steps.at(-1)?.settled === true ? Option.some(yield* Fiber.join(fiber)) : Option.none();
yield* Fiber.interrupt(fiber);
return { steps, report };
}).pipe(
Effect.provide(Layer.mergeAll(Live, Registry.platformLayer, TestClock.layer())),
Effect.scoped,
),
);
expect(Exit.isSuccess(exit)).toBe(true);
if (Exit.isFailure(exit)) return;
// Still waiting a millisecond before thirty seconds; the first request is
// cut short at thirty, the second thirty after.
expect(exit.value.steps).toEqual([
{ settled: false, interrupted: 0 },
{ settled: false, interrupted: 1 },
{ settled: true, interrupted: 2 },
]);
const note = "no answer within 30s, the run's `requestTimeout`";
expect(
Option.map(exit.value.report, (report) => outcomeFor(report, "spec/target/reachable")),
).toEqual(
Option.some(
expect.objectContaining({
_tag: "Failed",
findings: [
expect.objectContaining({ observed: `no response (${note})` }),
expect.objectContaining({ observed: `no response (${note})` }),
],
}),
),
);
});

it.each(["Infinity", Duration.infinity] as const)(
"waits as long as the transport does when the run turns it off with %s",
async (requestTimeout) => {
const exit = await clocked({ requestTimeout }, ["30 seconds", "1 hour"]);
expect(Exit.isSuccess(exit)).toBe(true);
if (Exit.isFailure(exit)) return;
const run = exit.value;
expect(run.settled).toEqual([false, false]);
// Only the first request was sent; it was still waiting.
expect(run.interrupted).toBe(1);
},
);

it.each([0, -1, Number.NaN, "-Infinity", "1e3 seconds"] as const)(
"dies on a `requestTimeout` of %s before it sends a request",
async (requestTimeout) => {
let sent = 0;
// The client answers every request at once, so a run that sent one
// before reading its options still finishes, and fails on `sent`
// rather than hanging.
const exit = await Effect.runPromiseExit(
Conformance.verifyWith(api, {
client: HttpClient.make((request) =>
Effect.suspend(() => {
sent += 1;
return Effect.fail(
new HttpClientError.HttpClientError({
reason: new HttpClientError.TransportError({ request, description: "reset" }),
}),
);
}),
),
baseUrl: "http://fixture.test",
ids: { articles: "1" },
only: ["spec/target/reachable"],
requestTimeout,
}),
);
expect(Exit.isFailure(exit) && Cause.hasDies(exit.cause)).toBe(true);
expect(Exit.isFailure(exit) && String(Cause.squash(exit.cause))).toContain(
"`requestTimeout` must be a positive duration",
);
expect(sent).toBe(0);
},
);

it("changes nothing for a server that answers within it", async () => {
const report = await verifyCanned(conformantServe, { requestTimeout: "1 minute" });
expect(failed(report)).toEqual([]);
expect(report.outcomes).toEqual((await verifyCanned(conformantServe)).outcomes);
});
});

// ---------------------------------------------------------------------------
// The control: a conformant server
// ---------------------------------------------------------------------------
Expand Down Expand Up @@ -2842,26 +3118,34 @@ describe("Conformance: the write fixture", () => {
expect(report.outcomes.filter((outcome) => Predicate.isTagged(outcome, "Errored"))).toEqual([]);
});

it("lets a client retry a request whose preparation failed, as its own `execute` does", async () => {
const direct = await run();
const failedOnce = new WeakSet<HttpClientRequest.HttpClientRequest>();
let refused = 0;
const { report } = await run(undefined, undefined, {
client: (canned) =>
HttpClient.retry(
HttpClient.mapRequestEffect(canned, (request) => {
if (failedOnce.has(request)) return Effect.succeed(request);
failedOnce.add(request);
refused += 1;
return Effect.fail(unprepared(request));
}),
{ times: 1 },
),
});
expect(refused).toBeGreaterThan(0);
expect(failed(report)).toEqual([]);
expect(verdicts(report)).toEqual(verdicts(direct.report));
});
// The timeout wraps the client's whole `execute`, so a `retry` runs within it
// and the request still runs through the client exactly as before.
it.each([
["", undefined],
[", within a `requestTimeout`", "1 minute"],
] as const)(
"lets a client retry a request whose preparation failed, as its own `execute` does%s",
async (_, requestTimeout) => {
const direct = await run();
const failedOnce = new WeakSet<HttpClientRequest.HttpClientRequest>();
let refused = 0;
const { report } = await run(undefined, () => ({ requestTimeout }), {
client: (canned) =>
HttpClient.retry(
HttpClient.mapRequestEffect(canned, (request) => {
if (failedOnce.has(request)) return Effect.succeed(request);
failedOnce.add(request);
refused += 1;
return Effect.fail(unprepared(request));
}),
{ times: 1 },
),
});
expect(refused).toBeGreaterThan(0);
expect(failed(report)).toEqual([]);
expect(verdicts(report)).toEqual(verdicts(direct.report));
},
);

it("names a redirect the transport followed off the origin in the rule it leaves undecided", async () => {
const unknown = await run(undefined, undefined, {
Expand Down Expand Up @@ -3222,6 +3506,8 @@ describe("Conformance: the write fixture", () => {
ids: { articles: "1" },
mutating: true,
writes: { articles: fixture(store, cleaned) },
// The PATCH must still be waiting when the interrupt comes.
requestTimeout: "Infinity",
}),
);
yield* Deferred.await(patching);
Expand Down
Loading
Loading