Skip to content

Resolve a page's relative request against its base, as the browser does - #1564

Open
rudycelekli wants to merge 1 commit into
CopilotKit:mainfrom
rudycelekli:fix/network-hook-base-uri
Open

rudycelekli wants to merge 1 commit into
CopilotKit:mainfrom
rudycelekli:fix/network-hook-base-uri

Conversation

@rudycelekli

Copy link
Copy Markdown
Contributor

What this changes

In page mode, the Recorder can record a request at an address the page never called.

When Chrome refuses the debugger, the Recorder falls back to page mode and injects extension/src/main-world/network-hook.ts into the page. The hook wraps fetch and XMLHttpRequest.open and records each request's address with absolute(url), which is new URL(url, location.href). The browser resolves a relative address against document.baseURI instead, and a <base href> element moves that away from the page's own address. On a page with one, the recorded url, route and origin name a request that was never made.

Example: on https://shop.example.com/app/orders/42 with <base href="/">, which Angular apps set by default and whose HttpClient calls are often written as api/..., fetch("api/orders/42/archive") goes to https://shop.example.com/api/orders/42/archive. The hook recorded https://shop.example.com/app/orders/api/orders/42/archive. A <base href> on another origin gets the origin wrong too.

That value is the step's network exchange. It feeds the step's save request, the repeat signature, and matchRun / matchRuntimeInfoRequest, so the recording describes an endpoint that doesn't exist. In debugger mode CDP reports the real URL, so the same page records different addresses depending on which mode Chrome allowed.

The fix resolves against document.baseURI, which is what fetch() and XMLHttpRequest.open() use. A page without <base> has document.baseURI === location.href, so nothing changes for it.

Where it runs

OpenBot is deployed as several server processes behind a load balancer, serving a whole company.
Consecutive requests from the same person reach different processes, and the process that answered a
WebSocket upgrade is rarely the one that answers the next call on that conversation.

State that outlives a single request therefore has to be shared, or the change works on one machine
and stops working the moment there are two, without saying so. That failure is worse than not
shipping the feature: it passes review, passes CI, passes a local demo, and only surfaces as a Bot
that forgets, a question nobody can answer, or a boundary that never fires.

Answer these even when the answer is "none":

  • New state that outlives a request? None. This runs in the recorded page's main world and only changes how one address is resolved.
  • What happens on the second replica? No change. The server isn't involved.
  • Anything serialised? No.
  • Anything fanned out to a browser? No.
  • New listener, port, or schedule? No.

Postgres is already there and is the default answer to all of the above: a table, a unique index, a
conditional update, LISTEN/NOTIFY for fan-out.

Boundary and audit

  • Every acting call still goes through the gateway: resolve, decide, audit, then act. Unchanged. This is the Recorder's capture, not an acting call.
  • New refusals and new failures each write a row. There are none.
  • Nothing new is trusted from the client that the server can resolve itself. Unchanged. Masking runs on the event as before, in the page and again in the service worker.

Changelog

  • A line in CHANGELOG.md under Unreleased, at the top of Smaller changes:
  • The Recorder records a page's requests at the address the browser sends them to. When Chrome
    would not let the Recorder use its debugger, it watched requests from inside the page and
    resolved a relative address against the page's own address, though a browser resolves it against
    the page's <base href>. On a site that sets one, such as many Angular apps, a step's request was
    recorded at an address that was never called. It is now the address the debugger reports.

Proof

New test in extension/tests/page-network-hook.test.ts, a relative address is recorded where the browser sends it, under the page's <base href>. The happy-dom page at /page gets <base href="/api/">. Its fetch resolves against document.baseURI the way a browser's does. fetch("small") must then be recorded with route /api/small and url http://127.0.0.1:<port>/api/small.

Rebuilt on main at f8af37ab. With only the source change reverted (git checkout origin/main -- extension/src/main-world/network-hook.ts), the new test fails:

$ bun test ./extension/tests/page-network-hook.test.ts -t "base href"
(fail) a relative address is recorded where the browser sends it, under the page's <base href> [61.86ms]
0 pass
1 fail

With the fix:

$ bun test ./extension/tests/page-network-hook.test.ts
17 pass
0 fail

Gates: bunx biome format and bunx biome lint --error-on-warnings on the touched files report nothing. cd extension && bun run typecheck (tsc --noEmit) exits 0.

The page network hook resolved fetch() and XMLHttpRequest.open() addresses
against location.href. The browser resolves them against document.baseURI,
which a <base href> moves, so on such a page the recorded url, route and
origin named an address that was never requested.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant