Skip to content

ios: text-entry lanes fail intermittently on unrelated branches (unpinned simulator keyboard state) #1874

Description

@thymikee

Status (2026-08-26). Both filed symptoms are fixed and merged (#2035, 7db5ad73dd). One of
them is genuinely root-caused; the other is mitigated. This issue stays open for the mitigated
half — why the simulator input pipeline throttles at all — which is the needs-info the label
refers to. The original "Root cause" and "Fix" sections below were wrong in both of their
mechanisms; they are kept, marked, under Superseded hypotheses, with the
measurements that disproved them.

The targeted iOS XCUITest and the fixture-backed iOS smoke both fail intermittently on branches that touch no Swift and no iOS production code, blocking the most expensive PR gate.

Two symptoms — and, it turned out, two unrelated causes:

  • RunnerTests.testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden — ** TEST EXECUTE FAILED ** / exit 65
  • smoke:form-input → step fill full name → TEXT_INPUT_COMMIT_NOT_OBSERVED
run branch when
32229709270 ci/1824-coverage-fork-death (#1854 — test-setup, docs, vitest configs only) 2026-08-19
32176591415 agent/fix-install-source-network-cause (#1863) 2026-08-18
32847092583 (both attempts) claude/mcp-registry-security-el547d (#2023 — MCP/release only, 16 files) 2026-08-25
32655577032 (attempt 1 only; attempt 2 green on the same SHA) claude/monorepo-modularity-analysis-t1kd3c (#1985 — one TS module renamed, three imports retargeted) 2026-08-23
32889322172 claude/1874-text-entry-commit-budget (#2035, this issue's own fix branch — the run that produced the decisive trace) 2026-08-25

Root cause

smoke:form-input — resolved, and it was never a race

The fixture asked fill for the one value it cannot verify. field-name's placeholder was
"Ada Lovelace", and every checkout-form suite filled exactly "Ada Lovelace"; field-email had
the same collision with ada@example.com. Twelve fills across eight files.

An empty text field renders its placeholder as its accessibility value, so element.value is
byte-identical whether the placeholder is showing or the committed text happens to equal it, and
placeholderValue does not disambiguate. awaitSynthesizedReplacementCommitOutcome therefore
refuses such a wait outright (textMatchesPlaceholder) rather than reporting a commit it cannot
prove — the documented contract, see the TEXT_INPUT_COMMIT_NOT_OBSERVED paragraph in
website/docs/docs/commands.md and the three-state model in #1924.

Caught from run 32889322172,
where the trace is unambiguous:

[DEBUG-1874] wait start expectedLen=12 route=replacement          19:35:00.652
   (no poll lines at all)
[DEBUG-1874] wait outcome=notObserved elapsedMs=3608 route=replacement   19:35:04.260

Zero poll lines: the wait never entered its loop. expectedLen=12 is "Ada Lovelace". The 3.6s is
the AX resolution for the placeholder read that precedes the guard, not a deadline.

Why it looked intermittent. The synthesized-replacement route is gated on
xCTestChannelPenalized — it is taken only when the host is loaded. A deterministic refusal on a
load-gated route is indistinguishable from a load flake from the outside. It is also what actually
explains the datapoint this issue found most puzzling: rerun_failed_jobs on 32847092583
reproducing identically. That is not persisted per-device state; it is the same guard firing on
the same input.

Fixed at the fixture (placeholders changed), not at the filled values — frozen
test/replay-compat/scripts/** corpora carry the same fills and must not be edited. The collision
had also been making the read-back assertions vacuous: assertJsonContains(name, 'Ada Lovelace') is
satisfied by an empty field rendering the placeholder.

testBareType… — mitigated, not root-caused

The commit wait used a flat 3s deadline, which cannot tell a throttled input pipeline
(characters keep landing, slowly) from a wedged one (nothing lands). It condemned both at the
same instant, so a working type reported TEXT_INPUT_COMMIT_NOT_OBSERVED.

The wait now grants time against progress — the observed value's expected-prefix growing, the same
length-only evidence logCommitCadence already emitted — with an absolute 10s ceiling. A pipeline
making no progress still expires at exactly the 3s the flat deadline used, so nothing that failed
before starts passing merely by waiting longer.

This reduces exposure; it does not explain the throttle. The ~40s episode traced in the
2026-08-21 diagnosis comment below would still exhaust the ceiling and report honestly.

Superseded hypotheses

Both mechanisms originally filed here are disproven. Recording them so they are not re-derived:

1. "Nothing pins the simulator keyboard state." Measured on a dedicated simulator, headless
simctl boot, iOS 26.2, then deleted:

per-device ConnectHardwareKeyboard Settings search field, after tap
true (hardware keyboard "connected") full software keyboard
false full software keyboard

com.apple.iphonesimulator is Simulator.app's preference domain; with no Simulator.app running,
CoreSimulator does not consult it. Pinning it would have added a step, a guard test and a green
assertion proving only that defaults write ran. The useful half of the finding is the reassuring
one: a headless lane has no hardware keyboard to inherit, so the ambient state named in this
issue is not a variable in these lanes at all.

2. "Text-entry readiness depends on that ambient state." Neither failing signature reaches
waitForTextEntryReadiness:

  • testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden types through the tap witness, and
    rememberedTextEntryTarget() returns before readiness is ever consulted.
  • smoke:form-input's fill fails inside runSynthesizedReplacementRoute, which
    shouldUseSynthesizedFirstResponderReplacement gates on hasResolvedElement == false — the
    penalized coordinate route builds that target directly in executeTypeCommand, bypassing
    focusTextInputForTextEntry entirely.

The readiness fallback was hardened anyway (it now returns only on confirmed focus of the target,
re-arming otherwise), because the latent sensitivity described here was real — it just was not what
was firing.

What shipped

#2035 (7db5ad73dd):

  • progress-aware commit budget (SynthesizedCommitDeadline), stall budget unchanged at 3s, ceiling 10s;
  • readiness's hardware-keyboard fallback returns only on confirmed focus of the target element;
  • testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden's keyboard-hidden precondition is
    XCTSkipIf, not XCTAssertFalse — an environment flip must not read as a product regression;
  • the two fixture placeholders.

What is still open

Why does the simulator input pipeline throttle? Unchanged by the above. What is different is
that the evidence to answer it is now on the shipped path rather than needing new instrumentation:
the [DEBUG-1874] per-poll cadence discriminates three shapes, and after #2035 the same
expected-prefix signal decides the deadline rather than merely being logged.

cadence shape meaning
prefix grows slowly, then completes throttle — absorbed by the budget, no red at all
prefix freezes wedge — condemned at the 3s stall budget, reported honestly
no poll lines the placeholder guard fired before polling (unreachable in this fixture now)

So the next red lane should be readable from runner.log in the uploaded ios-artifacts bundle
without adding anything. 1874-diagnose.yml (matrix isolated/pair × x86_64/arm64) remains
dispatchable for forcing the question rather than waiting for it.

Related: #1924 (honest reporting, and the three-state placeholder model), #1941 (commit-wait
instrumentation), #1995 (fill's synthesized-replacement verification), #2032 (folded in here),
#2053 (a different Coverage-lane flake seen while working on this, not this issue).

Activity

  1. 15 remaining items

  2. thymikee commented on Aug 26, 2026

    @thymikee
    MemberAuthor

    Body rewritten to match what actually landed (#2035, 7db5ad73dd). Summary of the corrections, since the previous text would have sent the next reader down two dead ends:

    The two symptoms had two unrelated causes, not "one cause" as filed.

    smoke:form-input is root-caused and fixed, and it was never a race: field-name's placeholder was "Ada Lovelace" and every checkout-form suite filled exactly "Ada Lovelace" (same for field-email / ada@example.com) — twelve fills across eight files. An empty field renders its placeholder as its AX value, so the commit wait refuses before polling rather than claiming a commit it cannot prove. It only surfaced under load because the synthesized-replacement route is gated on xCTestChannelPenalized. That also explains the datapoint this issue found most puzzling — rerun_failed_jobs reproducing identically on 32847092583. That was not persisted per-device state; it was the same guard firing on the same input.

    testBareType… is mitigated, not root-caused: the flat 3s deadline could not tell a throttled pipeline from a wedged one, and now grants time against progress with a 10s ceiling.

    Both originally filed mechanisms are disproven, and are kept in a "Superseded hypotheses" section with the measurements rather than deleted. Pinning ConnectHardwareKeyboard is a no-op under headless simctl boot (A/B table in the body) — with the reassuring corollary that a headless lane has no hardware keyboard to inherit, so the ambient state this issue is named for is not a variable in these lanes. And neither failing signature reaches waitForTextEntryReadiness; the readiness fallback was hardened anyway, because that latent sensitivity was real, just not what was firing.

    What the issue is now open for is stated explicitly, with the discriminator: the [DEBUG-1874] cadence tells throttle (prefix grows, absorbed) from wedge (prefix freezes, condemned at 3s) from the placeholder guard (no poll lines at all), so the next red lane should be readable from the uploaded runner.log without adding instrumentation.

    The occurrence table gained the two runs recorded in comments but missing from it, including 32889322172 — this issue's own fix branch, which is where the decisive trace came from.

    I have dispatched 1874-diagnose.yml on main (25 iterations × isolated/pair × x86_64/arm64) to try to force a cadence trace post-fix rather than wait for one: run 32983174128. Results to follow.


    Generated by Claude Code

  3. thymikee commented on Aug 26, 2026

    @thymikee
    MemberAuthor

    Post-fix diagnose dispatch: 100/100 green, and the residual slowness is before the wait

    Run 32983174128 on main at 7db5ad73dd — 25 iterations × isolated/pair × x86_64/arm64.

    leg verdict min median p90 max
    arm64 isolated 25 passed, 0 stalled 654 ms 796 ms 1131 ms 14146 ms
    arm64 pair 25 passed, 0 stalled 628 ms 702 ms 807 ms 1106 ms
    x86_64 isolated 25 passed, 0 stalled 927 ms 1175 ms 1790 ms 2581 ms
    x86_64 pair 25 passed, 0 stalled 908 ms 1180 ms 1542 ms 1765 ms

    durationMs is the type-all phase — the whole synthesized dispatch, not just the wait.

    Three things worth recording.

    1. The loop no longer reproduces the failure. 100 iterations, zero stalls, on the same harness and hardware class that produced the filed episodes. That is consistent with #2035 absorbing them, and it is not evidence the throttle is gone — see (3).

    2. Rosetta costs ~40% at the median (arm64 796/702 ms vs x86_64 1175/1180 ms), independently corroborating the ~30% figure that motivated the AGENT_DEVICE_XCUITEST_ARCHS=arm64 pin in #1941.

    3. Every slow outlier settled on the FIRST poll. Five iterations exceeded 2 s across the two isolated legs — 14146, 4021, 2581, 2403, 2245 ms — and all five report polls=1. The commit wait observed the text on its first look; the seconds went somewhere before it, in the synthesize post or the element resolution ahead of it. That reframes this issue's open question: not "why does delivery throttle mid-burst", but "what blocks before the first observation".

    A related shape, offered as hypothesis rather than conclusion (n is small): both outlier-carrying legs are isolated; neither pair leg has a single outlier, on either arch. pair adds testBareDelayedTypeFailsWhenTappedInputDisappearsMidCommand, which sorts first and so launches the fixture app before the measured test runs. Cold input/AX path, warmed by the neighbour, would explain both that and the iter=1 outlier present in each isolated leg.

    The dispatch also proved the harness was not collecting the evidence

    Zero [DEBUG-1874] cadence lines and zero per-iteration logs came back — the artifacts are four stall-summary.txt files and nothing else. The harness dumped cadence only for failing iterations, and since #2035 an absorbed episode passes. So a fully green run, containing a 14-second outlier, produced no trace of it.

    #2059 fixes that: cadence is captured for any iteration slower than 2000 ms (a threshold measured from the 100 healthy iterations above), and the classifier reads xcodebuild's own verdict word instead of counting every non-passed result as a stall — which matters now that #2035 gave the looped test an XCTSkipIf.

    Next step for closing this issue: re-dispatch after #2059 lands. If the outliers reproduce with cadence attached, the synthesize posted … tookMs line should localize the pre-poll cost, which is the needs-info this is labelled for.


    Generated by Claude Code

  4. thymikee commented on Aug 27, 2026

    @thymikee
    MemberAuthor

    The dispatch answers it: the input path is fast, the accessibility read path is what stalls

    Run 33072911858 — 25 iterations × isolated/pair × x86_64/arm64, on the fixed harness (#2059), which unlike the previous dispatch captures cadence for slow passes.

    100/100 passed again, 0 skipped, 0 failed, 0 no-result. Distributions:

    leg median p90 max
    arm64 isolated 827 ms 1076 ms 1869 ms
    arm64 pair 842 ms 896 ms 1085 ms
    x86_64 isolated 1083 ms 2042 ms 2166 ms
    x86_64 pair 1407 ms 2427 ms 14334 ms

    The 14.3 s iteration, with its cadence attached for the first time:

    [DEBUG-1874] synthesize posted 17 chars status=0 tookMs=484
    [DEBUG-1874] wait start expectedLen=17 route=append
    [DEBUG-1874] poll t=440ms observedLen=17 expectedPrefixLen=17
    [DEBUG-1874] wait outcome=settled elapsedMs=1229 route=append
    

    Posting the 17 characters took 484 ms. The commit was observed on the first poll, and the whole wait was 1229 ms. Yet type-all — which brackets typeIntoCurrentTarget — measured 14334 ms.

    So roughly 12.6 s went to neither the synthesis nor the commit. What is left inside that bracket is resolveTextEntryElement and editableTextValue: accessibility round-trips, before a single character is posted.

    That reframes this issue's premise. It is titled and was investigated as an input-delivery throttle. On this evidence the input pipeline is not throttled at all — 484 ms to post, first-poll commit. What stalls is the AX read path, which independently matches the 2026-08-21 note that "during stall windows everything touching the input system is slow (AX reads ~12s)". Those AX reads are the same ones #1105 tracks.

    Caveat, and a defect this dispatch exposed: the per-iteration logs were not in the artifact, so this rests on the 40-line cadence excerpt rather than the full log. .tmp is a hidden directory and upload-artifact skips those unless told to include them — so every per-iteration log this workflow has ever kept was silently discarded. Fixed in #2059 (include-hidden-files: true); the next dispatch will carry the intervening lines, which would let the 12.6 s be attributed to a specific AX call rather than to the pair of them.

    Proposed disposition

    Both filed symptoms are resolved: smoke:form-input root-caused and fixed (the fixture placeholder collision), and the targeted XCTest mitigated by the progress-aware budget, with 200 consecutive green loop iterations across two dispatches. The remaining slowness is real but is an AX-read cost, not the delivery throttle this issue is named for — and it no longer reddens lanes.

    I would close this as mitigated, with the throttle premise corrected, and let the AX-read cost be pursued under #1105 where that path is already tracked. #2080 carries the one live correctness defect found along the way.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    needs-infoWaiting on reporter or external input

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions