diagnose(1874): instrument the synthesized commit wait and add a dispatchable stall loop

[thymikee]

Refs #1874, #1844. Follow-up to #1924 (merged): production now reports TEXT_INPUT_COMMIT_NOT_OBSERVED instead of ok: true over a partial commit. This PR ships the diagnosis tooling for the remaining question — why synthesized-event delivery occasionally stalls on CI.

What we know (evidence from 2026-08-18..21 failures)

• The stall is delayed delivery, not dropped events: run 32176591415 typed "hardware-keyboard", held "ha" at failure time, and the field read "hardware-key" ~40s later with nobody typing. The burst drains late.
• During stall windows everything touching the input system is slow: AX value reads took ~12s, subsequent focus took 2.6–13s.
• CI runs the test bundle as x86_64 under Rosetta on M1 Virtual, 3-core, 7 GB runners (macos-26-arm64 image; Xcode 26.6 builds x86_64 from -destination generic/platform=iOS Simulator).
• Not reproduced by: local arm64/x86_64 runs, 10-core synthetic load, Rosetta on iOS 18.6, or running after the disappearing-input predecessor.
• A 50-iteration instrumented loop on the real CI image (run 32485228371) passed 50/50 but caught one sub-critical episode: type-all 2487ms across 5 polls — the same mechanism at recoverable severity. Episodes that outlast the 3s commit window are the rare failures (~4 in days of full-suite runs).

Conclusion: transient throttling of the synthesized input pipeline on CI VMs; most episodes recover inside the commit window, the tail does not.

What this PR adds

1. [DEBUG-1874] instrumentation in awaitSynthesizedFirstResponderCommit: per-poll timestamp, observed length, and expected-prefix length through the real injected observe closure, plus post time of the synthesize call and the final outcome — value-free by construction (typed Int-only boundary, sentinel-tested). Any future red lane now carries a cadence trace instead of a bare value mismatch, and no typed field content reaches runner.log.
2. .github/workflows/1874-diagnose.yml — workflow_dispatch matrix (isolated vs pair-after-predecessor) looping the bare-type test N times on the CI image, with per-iteration verdicts and stall evidence artifacts.

Validation

• pnpm gate swift-runner-ios — ** TEST BUILD SUCCEEDED **; local run prints [DEBUG-1874] poll ... outcome=settled elapsedMs=34.
• pnpm check:affected --run — all runnable checks passed.
• The workflow itself executed twice on this branch (runs 32479566150, 32485228371); the second completed 50/50 green with evidence artifacts.

Intent

Keep the workflow dispatchable until #1874's underlying stall is root-caused; when a lane goes red again, its logs will show whether the prefix walks slowly (delivery throttling) or freezes (queue wedge). Remove both once resolved.

Mitigation shipped in this PR

The 2×2 matrix (runs 32485228371, 32497125475) measured native arm64 at ~30% faster commits than Rosetta x86_64 on identical CI hardware (mean type-all ~860ms vs ~1150ms), with slow-delivery episodes occurring on both architectures. Rosetta amplifies episode severity; it is not the fundamental cause. This PR pins AGENT_DEVICE_XCUITEST_ARCHS=arm64 across every lane that builds the iOS runner (ios.yml, xctest-nightly, perf-nightly, conformance-differential, replays-manual) via the new build-script override, and bumps derived-data cache suffixes. TEXT_INPUT_COMMIT_NOT_OBSERVED (#1924) remains the backstop for episodes that outlast the commit window.

Cleanup ownership

This instrumentation and the dispatchable workflow are temporary. Removal trigger: #1874 (reopened — the delivery throttle is not root-caused) identifies the mechanism, at which point the [DEBUG-1874] boundary, its guard tests, and 1874-diagnose.yml are deleted in one commit. Owner: the iOS runner maintainers; the reopened issue is the tracking thread.

[github-actions]

Size Report

Metric	Base	Current	Diff
JS raw	2.36 MB	2.36 MB	0 B
JS gzip	791.7 kB	791.7 kB	0 B
npm tarball	911.8 kB	912.6 kB	+807 B
npm unpacked	3.16 MB	3.16 MB	+2.6 kB

npm unpacked components

Component	Base	Current	Diff
JS / dist source	2.51 MB	2.51 MB	0 B
Apple runner source/project	547.4 kB	550.0 kB	+2.6 kB
macOS helper source	54.5 kB	54.5 kB	0 B
Android helper artifacts	0 B	0 B	0 B
Other package files	44.4 kB	44.4 kB	0 B

Startup median (7 runs, lower is better):

Scenario	Base	Current	Diff
CLI --version	27.6 ms	27.3 ms	-0.3 ms
CLI --help	78.7 ms	78.9 ms	+0.3 ms

Top changed chunks: no changes in the largest emitted chunks.

Top changed packed files

Packed file	Base	Current	Diff
dist/apple/runner/AgentDeviceRunner/AgentDeviceRunnerUITests/RunnerTests+SynthesizedTextEntry.swift	10.7 kB	13.3 kB	+2.6 kB

[thymikee]

P1 — the diagnostic instrumentation writes raw observed field contents to NSLog on the shipped synthesized-text path: observedText.prefix(40). This is not confined to the workflow or unit-test build, so ordinary type commands can persist credentials/PII in runner.log; the diagnostic workflow then uploads those logs on a stall. Please keep the cadence evidence value-free (for example observed length, expected-prefix match length, progress state, and timestamps), or gate any raw fixture-only trace behind a compile-time diagnostic build that cannot ship. Add a regression/source guard that makes reintroducing raw observed text into runner logging fail.

[thymikee]

Addressed in 634367f. The per-poll trace now logs only t, observedLen, and expectedPrefixLen — lengths and the expected-prefix walk, no field contents; same evidence for discriminating throttle (prefix grows) from wedge (prefix freezes). Every string-interpolating NSLog format in the module is allowlisted by src/__tests__/apple-runner-log-redaction.test.ts, which fails on any new %@ format — reintroducing raw observed text requires editing that allowlist, i.e. a reviewed change. Guard verified red against the pre-fix revision (unreviewed string-interpolating log format "[DEBUG-1874] poll t=%.0fms observed=%@") and green after; commonPrefixLength is pinned in the host-lane policy tests. The diagnostic workflow's uploaded logs therefore contain no typed field values from this path.

[thymikee]

P1 — The shipped poll path is value-free now, but the new guard does not enforce that invariant. It only reviews literal NSLog formats containing %@, so Swift interpolation of observedText is skipped entirely, and passing observedText to an already-allowlisted %@ format also passes. Please put cadence logging behind a typed value-free boundary that accepts only lengths, timestamps, and enums, then test its emitted output with a sentinel secret instead of partially parsing Swift format strings. Update the PR body too: item 1 still says the trace includes the observed value, which is no longer true. Finally, #1874 and #1844 are both closed while the PR says this temporary instrumentation remains until #1874 is root-caused; reopen the cleanup issue or name an active owner and removal trigger before shipping it.

[thymikee]

Addressed in e7524f9.

1. Typed boundary: cadence logging now goes through logCommitCadence(elapsedMs:observedLen:expectedPrefixLen:) — Int-only parameters, so observed contents are unrepresentable at the poll call site. Its emitted line is a pure function (commitCadenceLogLine) pinned by a sentinel-secret test in the host-lane policy tests: exact-equality on the output for sentinel inputs, asserting no content channel exists.
2. Guard: rewritten as structural — boundary signature present, poll path logs through it, zero raw NSLog inside the observe closure. No Swift format-string parsing. Verified red against the pre-boundary revision, green after.
3. PR body updated (item 1 now describes value-free evidence) and ios: testBareTypeUsesTappedInputWhenSoftwareKeyboardIsHidden fails intermittently on unrelated branches (blocking ios.yml PR gate) #1874 reopened as the removal-tracking thread: the PR names it as owner of the cleanup — instrumentation, guard tests, and the workflow go in one commit once the throttle mechanism is identified.

[thymikee]

Not merge-ready on the current base yet: #1948 landed after this branch and changes the same synthesized-commit wait from TextEntryTiming.pollInterval to TextEntryTiming.synthesizedCommitPollInterval (200 ms) in RunnerTests+SynthesizedTextEntry.swift. The merge is textually clean and Git's combined result preserves the new sparse interval, but every #1941 check and the arm64/x86 stall evidence ran against the old polling cadence—the behavior this diagnostic is meant to measure.

Please rebase onto current main, verify the combined wait still calls logCommitCadence(...) and sleeps with synthesizedCommitPollInterval, then rerun the iOS/host checks and the diagnostic matrix. No remaining code/privacy finding at e7524f9b8; the typed value-free boundary, sentinel test, active #1874 cleanup owner, and arch cache-key changes look sound.

[thymikee]

Rebased onto current main (956cd37, includes #1948). The combined wait verified: waitForNextObservation sleeps on TextEntryTiming.synthesizedCommitPollInterval (200 ms, with #1948's sparse-read rationale comment intact) and the observe closure logs through logCommitCadence(...) — both changes compose without conflict.

Reruns: both Swift gates build locally (** TEST BUILD SUCCEEDED **), local run shows the boundary-formatted trace over the sparse interval, check:affected green. The diagnostic matrix is rerunning on the new base (run 32556515431) and the PR checks are executing the host/iOS lanes against the rebased head.

[thymikee]

Re-reviewed at 956cd375 after the #1948 rebase: the combined wait preserves the sparse 200 ms synthesizedCommitPollInterval and routes cadence evidence through the Int-only value-free logging boundary. Exact-head iOS smoke, Host XCTests, coverage, integration, and static gates pass; remaining diagnostic matrix legs are still in progress. Code review is clean. LGTM.

[github-actions]

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-08-22 11:39 UTC