fix(ci): test failures caused by runner downgrade — blacksmith-4vcpu vs ubuntu-latest

[terisuke]

Root Cause Analysis

Symptom

Two tests consistently fail in our fork's CI but pass in upstream:

1. prompt submitted during an active run is included in the next LLM input
2. hook.execute > runHook > timeout returns pass

Root Cause: Runner Performance Degradation

	Upstream	Our Fork
Runner	blacksmith-4vcpu-ubuntu-2404	ubuntu-latest
vCPUs	4 (dedicated Blacksmith)	2 (shared GitHub)
Trigger	pull_request	pull_request_target

The upstream CI uses Blacksmith 4-vCPU dedicated runners which provide consistent, fast execution. Our fork switched to ubuntu-latest in PR #64 to resolve fork CI permission issues (pull_request_target).

The two failing tests are timing-sensitive:

• prompt-during-run: relies on async message persistence happening within tight windows
• hook timeout: sleep 10 + 200ms hook kill + process cleanup must complete within test harness timeout

On 4-vCPU dedicated runners, these timing windows are met. On 2-vCPU shared runners, they are not.

Evidence

• Upstream v1.3.17 CI: ALL GREEN (Blacksmith runner)
• Our fork CI: consistent unit failures on same test code
• Local tests (M-series Mac): ALL PASS
• The test code itself is identical between upstream and fork

Fix Options

Option A (Recommended): Restore Blacksmith runners

• Change ubuntu-latest back to blacksmith-4vcpu-ubuntu-2404 in .github/workflows/test.yml
• Requires Blacksmith plan for the Cor-Incorporated org
• Resolves all timing issues without code changes

Option B: Increase timeouts (current workaround in PR #127)

• Hook timeout: 5s → 15s
• Prompt poll interval: 20ms → 50ms, inner timeout: 5s → 8s
• Effect.sleep barriers added before/after gate resolution
• Mitigates but doesn't fully resolve the prompt-during-run race

Option C: Skip timing-sensitive tests in CI

• Mark with test.skip when process.env.CI
• Not recommended — hides real regressions

Recommendation

Pursue Option A as the permanent fix. If Blacksmith isn't available for the fork, Option B is acceptable with the understanding that the prompt-during-run test may still intermittently fail on slow runners.

Related

• fix(ci): flaky unit tests — prompt-during-run and hook timeout #121 (flaky tests — originally misidentified as "pre-existing flake")
• fix(ci): use pull_request_target for fork runner compatibility #64 (fork CI runner migration to pull_request_target)
• PR fix(ci): resolve test failures on 2vCPU shared runners #127 (current workaround: timeout increases)

[terisuke]

Status update (2026-04-08):

• PR fix(ci): resolve test failures on 2vCPU shared runners #127 merged: timing tolerances widened for 2vCPU runners
• PR chore: sync upstream/dev (v1.4.0, tool refactor, OTLP, subagent UX) #136 (upstream sync v1.4.0) includes test: disable GPG signing in test fixtures (#20386) which may further improve CI stability
• Root cause (blacksmith 4vCPU vs ubuntu-latest 2vCPU) remains a structural limitation of the fork

No further action needed unless CI failures recur after #136 merge.

[terisuke]

Status update (2026-04-08): Still affects e2e/unit timing-sensitive tests. All tests pass locally. The 2vCPU ubuntu-latest vs 4vCPU blacksmith gap remains. No workaround available without Blacksmith subscription. Non-blocking — CI checks other than e2e/unit all pass consistently.

[terisuke]

Cross-reference

Related to #166 (persistent e2e flaky tests). Both issues share the same root cause: fork CI using ubuntu-latest (2vCPU) vs upstream blacksmith (4vCPU).

Proposed resolution: Address #129 and #166 together in a single Wave — increase timeouts in packages/app/e2e/actions.ts helpers and fix timing-sensitive test assertions.

Blocked: #149 (auto E2E tests) depends on CI stability from this fix.