[davidmckayv]

What

bun run test:e2e: Playwright against the real OpenBot, in a real browser, on a real local stack, with every model call answered by aimock in strict replay. Slice 1 of the end-to-end suite: the harness and the core loop. Runs on a developer's machine; CI does not run it (by design).

bunx playwright install chromium   # once
bun run test:e2e                   # strict replay
bun run test:e2e:record            # re-capture fixtures with the key in .env

The stack

Each run brings up its own fresh stack with scripts/start.sh (it never reuses one): personal edition (one person, no sign-in, the Personal Assistant), compose project openbot-e2e, its own image tags (so --build never replaces openbot-agent-computer:latest under another stack), ports 46xxx, a new database every run. A stack already running on the machine (OpenBot Desktop on 3001/3010/5432, a dev stack, another worktree) is not touched. E2E_KEEP_STACK=1 leaves it up afterwards.

scripts/start.sh gains two opt-in settings for that: OPENBOT_LOG_DIR (logs beside the screenshots, not over .logs/) and OPENBOT_PERSON_BROWSER=off (no Chrome window on the developer's desktop, as OpenBot Desktop runs).

Model traffic

All of it is OpenBot's own and goes to aimock: OPENAI_BASE_URL and ANTHROPIC_BASE_URL point at it, with dummy keys, before the server starts. On the core loop that is the Personal Assistant's run (POST /v1/responses), the skill chooser and the conversation's name (POST /v1/chat/completions), and the model list (GET /v1/models). models.dev capability data is switched off so the model stays the package's.

Not mocked because they are not model calls: Intelligence (licence, thread storage, the events the run is checked against). Automatic Learning, which would send conversations to Intelligence's own models, is not set up for the run.

Strict replay

• aimock runs strict, never proxy-only. A request no fixture answers fails the test that made it, naming it; after a run grep -c -E "NO FIXTURE MATCH|STRICT:" e2e/.artifacts/logs/aimock.log must print 0.
• Exact matching through aimock's requestTransform. By substring, the Bot's fixture for the person's question also answered OpenBot's conversation-naming call (which quotes the question), so the conversation was "named" with the Bot's reply and no naming fixture was ever recorded. That was a silent false pass; exact matching removes it.
• One normalisation: the naming call quotes the answer only if Intelligence has stored it by the time the 10 s naming loop runs, so the answer is cut from that call before matching and recording.
• Fixtures are recorded from the real product. Record mode hands the real key to aimock alone as its provider key; OpenBot holds an sk-aimock- dummy that aimock swaps on the way out. A recording containing the key is deleted and fails the test.

The core loop journey

Fresh stack, then as a person: Welcome, OpenBot for Business, name; Settings, Setup says the AI model needs a key and links to the Vault, where an OpenAI key is added (a dummy); a question to the Personal Assistant; the answer on screen is exactly what the run streamed, once; one run (one RUN_STARTED in the conversation's events, one run-model line in the server log, one model call); the conversation gets a generated name that is neither the question nor the answer. Every step saves a full-page screenshot to e2e/.artifacts/screenshots/core-loop/.

Run on a Mac

• Two strict replay runs back to back: both passed, 0 NO FIXTURE MATCH/STRICT lines, 3 of 3 model requests matched.
• Deliberate failure: the answer fixture changed to an unexpected tool call. The run failed loudly: 3 STRICT: No fixture matched for POST /v1/responses lines, the journey failed on the run not finishing and on the strict-replay check listing each missed request. Fixture restored, green again.
• All 11 screenshots read.

Best practice followed

• Playwright: role-based locators, web-first assertions and expect.poll, no fixed sleeps, isolated contexts (best practices); aimock as a webServer with a URL readiness check (web server); stack bring-up and teardown as a setup project with teardown, the documented alternative to globalSetup (global setup and teardown).
• aimock 1.48.0 (latest): strict mode and the journal (control API), record and provider keys (record and replay), exact userMessage matching under requestTransform (fixtures).

e2e/README.md covers running, recording, fixture scope and adding a journey.