Skip to content

iOS parameterized input remains in native XCTest diagnostic logs #3261

Description

@billsbooth

Observed behavior

On unmodified agent-device@0.21.22, a harmless value supplied through the Node API with fill({ text, recordAs }) was correctly absent from the fill response, subsequent predicate/error responses, and the inspected Agent Device session artifacts, but remained verbatim in the native XCTest .xcresult diagnostic log.

We are trying to use the official release for iOS login automation without carrying a downstream patch. Is there a supported way to prevent sensitive visible input from being retained by this native diagnostic channel? If this is outside the parameterization guarantee, documenting that boundary would also help.

Environment

  • Official npm release 0.21.22; source tag commit 202585240f0c1289642885b5df645cbc25114272
  • macOS host, Xcode 27.0 (27A266a), iOS Simulator
  • Stock signed XCTest runner; no modified upstream code
  • Script recording armed for recordAs; no screen recording, trace, or app-log capture requested
  • Only inert test strings were used, never a real credential

Reproduction outline

  1. Boot an owned iOS simulator and open Settings with the stock CLI in a fresh state directory/session:

    agent-device open com.apple.Preferences --platform ios --udid "$SIM_UDID" \
      --session stock-canary --save-script --foreground --timeout 120000 --json
  2. Use the official Node client for that same session. Take a fresh snapshot and select the unique Settings SearchField with label Search. Then execute:

    const options = { platform: 'ios', udid: process.env.SIM_UDID };
    const client = createAgentDeviceClient({ session: 'stock-canary', cwd: process.cwd() });
    // searchRef comes from the fresh Settings snapshot.
    await client.interactions.fill({
      ...options, ref: searchRef,
      text: 'INERT_STOCK_PRIVATE_CANARY_20261006',
      recordAs: 'PRIVATE_CANARY',
    });
    await client.interactions.is({ ...options, predicate: 'exists', selector: 'label="Settings"' });
    try { await client.interactions.press({ ...options, ref: '@e999999' }); } catch {}
  3. Inspect the exact runner's .xcresult Staging/1_Test/Diagnostics/.../Session-AgentDeviceRunnerUITests-*.log for the inert marker. Our observed excerpt:

    SearchField, label: 'Search', Keyboard Focused ... faulted in {
        "XC_kAXXCAttributeValue" = "INERT_STOCK_PRIVATE_CANARY_20261006";
    }: (no error) for value using XC_kAXXCAttributeValue
    

A second run using another inert marker and fill({ ..., verify: false }) also retained its marker in the native diagnostic log. The stock XCTest plan already sets system/user attachment lifetime to keepNever; that did not prevent this diagnostic text.

This establishes persistence during the sequence, not that the deliberately failed press alone caused it. We have not yet established the behavior on a physical iPhone or a masked password field. The visible Settings input was a control for a visible OTP field.

Expected / requested guidance

A documented stock path for sensitive visible input that avoids retaining its literal value in native diagnostics, or an explicit documented limitation so callers can stop before supplying credentials. We are not asking to disable ordinary observations globally.

Related: #1348 and #1398 provide useful recording/response redaction, and those protections worked in the checks above. #3260 separately concerns CLI input transport; it would not by itself address native diagnostics.

Activity

  1. billsbooth commented on Oct 6, 2026

    @billsbooth
    ContributorAuthor

    Additional bounded simulator control: we entered the inert digits 284690 using the stock coordinate-keyboard route with action recording disabled, then performed is exists 'label="Settings"' --no-record and a deliberately invalid-ref press with --no-record. No post-entry snapshot command was issued in this control. The same digits appeared in the native .xcresult diagnostic log.

    This means avoiding fill/typeText and using --no-record did not prevent native value persistence during this entry-and-completion-check sequence. We have not isolated which of the subsequent operations produces each native log line; we are not claiming every coordinate tap independently logs the complete value.

  2. billsbooth commented on Oct 6, 2026

    @billsbooth
    ContributorAuthor

    Confirmed on a physical iPhone as well, using the unmodified official 0.21.22 runner with automatic signing and a separate fresh task-owned state/session. The Settings Search control used another inert marker, INERT_STOCK_PHONE_CANARY_20261006, with the same fill({text, recordAs}) → stable predicate → invalid-ref error sequence.

    The inspected JSON/session outputs were redacted, but that exact marker appeared in the physical runner's .xcresult native diagnostic log. This extends the observed scope beyond Simulator. No real credential or masked-password field was tested.

  3. billsbooth commented on Oct 6, 2026

    @billsbooth
    ContributorAuthor

    Downstream scope update: our owner has clarified that these dedicated QA fixtures may appear in local test-device diagnostics. We are proceeding with the official unmodified release for that test-account workflow; this report is not a request to block ordinary automation or to maintain a downstream patch. The inert reproduction above remains valid, and documentation or supported handling of this native-log boundary would still be useful for callers whose inputs require stronger protection.

  4. thymikee commented on Oct 9, 2026

    @thymikee
    Member

    Decision: document the boundary now; prevention is separate research. Your simulator and physical-device controls show that XCTest's own diagnostic logs and the xcresult can contain typed text, and nothing in this repo controls them today. keepNever and coordinate entry don't change that.

    Next steps:

    1. Docs: state that the privacy guarantee for typed input covers agent-device's own output, logs and artifacts, and does not cover XCTest's native diagnostic logs or the xcresult on iOS.
    2. Research, keeping this issue open: whether the runner can stop XCTest from recording the parameter, or remove or redact the xcresult it owns, without breaking runner diagnostics.

    Moving secrets off argv is tracked in #3260.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    documentationImprovements or additions to documentation

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions