fix: probe isolation toolchain compatibility before timing

[janicduplessis]

Description

Refs #469 (does not close it: whether the pinned adapter is the accepted design, and whether a real untimed build/recording campaign check is still wanted on top of these probes, stays open there). The benchmark's sandbox-exec runner policy passes its filesystem probes but breaks two ordinary iOS operations, and nothing checked either before the clock started. This PR adds detection, not a new isolation design: dispatch refuses an iOS run before timing when the exact policy fails either probe and no native compatibility adapter is pinned, instead of failing inside the timed build or recording. Policy rules, grants, device ownership, version pins, the frozen campaign and published results are untouched.

The two operations:

• SwiftPM manifest and plugin sandboxes (the nested xcodebuild in the SDK 58 ExpoModulesJSI XCFramework build) fail with sandbox-exec: sandbox_apply: Operation not permitted.
• agent-device 0.20.10 reports simctl recordVideo did not expose a complete process identity because it reads the recorder's identity through /bin/ps, and the kernel refuses to exec /bin/ps from any sandboxed process.

No profile rule fixes the nesting. On macOS 26.5 only a profile that compiles to the same sandbox as the one already applied nests, and SwiftPM generates its own (deny default) profile per invocation, so no runner policy can match it; the pinned adapter from #483 (-IDEPackageSupportDisable*Sandbox=1, -disable-sandbox, the ps bridge) stays the only compensation.

Measurements: outer profile / nested profile / exit code

/usr/bin/sandbox-exec -p <outer> /usr/bin/sandbox-exec -p <inner> /usr/bin/true on macOS 26.5 (25F71). A profile that does not compile to the same sandbox as the outer one is refused in both directions, whether or not either profile contains a deny rule and even when the only difference is a rule with no effect; the same sandbox nests at any depth. swiftpm below is (version 1)(deny default)(import "system.sb")(allow file-read*)(allow process*), the head of the profile Sandbox.swift generates with per-invocation write grants; it exits 0 on its own.

outer	inner	exit
(allow default)	(allow default), with or without redundant allows	0
(allow default)	swiftpm	71
(allow default)	(allow default) plus one deny	71
exact runnerIsolationPolicy output	swiftpm, (allow default), or any other profile tried	71
(allow default) plus deny X	the same profile, also reordered, duplicated, or with redundant allows	0
(allow default) plus deny X	(allow default) plus deny Y, or plus deny X and Y	71
(allow default) plus deny X and Y	(allow default) plus deny X	71
swiftpm	swiftpm, also three levels deep	0
swiftpm	swiftpm plus one (allow file-write* ...), and the reverse	71
a deny plus (allow process-exec* (with no-sandbox) (literal "/usr/bin/sandbox-exec"))	(allow default)	0, but the nested child read the protected file: an escape, rejected

Solution

verifyIsolationCompatibility runs two untimed probes through isolatedRunnerInvocation, so they execute under the run's exact verified policy: a nested sandbox-exec applying the head of SwiftPM's profile, and /bin/sh -c '/bin/ps -p "$$" -o lstart=', the query agent-device makes for the recorder. The nested probe applies SwiftPM's (deny default) shape rather than a deny-free profile because only a profile that compiles to the same sandbox nests: a deny-free inner profile passes under any deny-free outer while SwiftPM's still fails there, so its permitted: true would not mean SwiftPM can build.

Dispatch calls it right after prepareRunIsolation, before the adapter probe, the profile smoke and the clock, and stores the record in the run's preflight.isolationCompatibility. It refuses only an ios run with no verified adapter (preflight.nativeCompatibility null); Android and adapter-pinned runs keep the record as evidence (Gradle does not nest sandboxes, Android recording reads /proc on the device, and the adapter's own probe proves the patched identity path). A refusal writes the record to isolation-compatibility.json in the run directory (driver) because a refused run never gets a meta.json; the error names the failing probe and its stderr and carries the record.

The probe cannot run earlier: the policy grants the run's fixture checkout, guidance file, STIM_HOME and tmp paths, which the setup before it creates, so a refused iOS dispatch still pays for the load gate and the golden copies, like every other refusal after the run directory exists.

Test plan

• scripts/agent-benchmark/runner-isolation.test.mjs, injected execute: every probe call goes to /usr/bin/sandbox-exec -p <exact policy> and the nested probe applies a (deny default) profile (catches a probe that runs outside the policy or with a deny-free inner profile and reports a false pass); the refusal message and the record attached to the error; no refusal with the adapter pinned or on Android; a clean record when both pass.
• Same file, real macOS (darwin-only, like the existing denial tests): a real verified policy yields sandbox-exec: sandbox_apply: Operation not permitted and /bin/sh: /bin/ps: Operation not permitted, iOS without the adapter throws, and the probe's own inner command run unsandboxed exits 0, so the refusal is the nesting rather than a broken invocation. The refusal assertions pin current kernel behavior on purpose: if Apple changes it, the adapter's premise changed and the test says so.
• Real tool calls (invariant 9) on macOS 26.5 (25F71): the matrix above; both probes with the driver's execFileSync options (status 71 and 126 with those messages under the exact profile); unpatched agent-device 0.20.10 host-process.js identity primitives under the exact profile: {"start":null,"command":null,"listError":"spawn EPERM"}.
• Not exercised: bench.mjs dispatch end to end (needs a benchmark root, pins, golden state and a runner), so the isolation-compatibility.json write is covered only by the unit test of the record on the error; a real simctl recordVideo (no simulator was booted, and I did not boot one); real xcodebuild package resolution and the full ExpoModulesJSI XCFramework build. The probes cover the mechanism each operation depends on, not the operation itself.
• pnpm test: 3704 passed; the 2 failures in engine-ios-device.test.ts are pre-existing on this machine (stale stim-devicectl-* directories in TMPDIR) and unrelated to this diff.

[janicduplessis]

Verdict: NEEDS CHANGES (finding 1 is actionable; GitHub refuses a request-changes review from the author account, so this is posted as a comment review).

Fresh review of the diff vs main, #469, the tests and the docs, with the sandbox claims re-run locally on macOS 26.5 (25F71).

Findings

1. The nested-sandbox probe cannot prove what it is named and documented as proving (scripts/agent-benchmark/runner-isolation.mjs:156, README.md:157-162, docs/agent-benchmark.md:111-113)

The probe nests a deny-free inner profile, (version 1)(allow default). SwiftPM's manifest and plugin sandbox is not deny-free: Xcode's swift-package carries /usr/bin/sandbox-exec, (deny default), (import "system.sb") (strings on Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-package). On this kernel a deny-containing inner profile is refused from any sandboxed process, deny-free outer profile included:

outer (allow default) / inner (allow default)                                      -> 0
outer (allow default) / inner (deny default)(import "system.sb")(allow file-read*)
                              (allow process*)(allow sysctl*)                       -> 71 sandbox_apply: Operation not permitted
outer (allow default) / inner (allow default)(deny file-read-data (literal ...))    -> 71
outer = runnerIsolationPolicy shape / same SwiftPM-shaped inner                     -> 71
same SwiftPM-shaped inner, not nested                                               -> 0  (profile is valid)
outer (allow default) / inner (allow default)(allow file-read*)(allow process*)     -> 0  (deny-free inner still nests)

Consequences:

• nestedSandbox.permitted: true (the value the darwin test asserts for an (allow default) profile at runner-isolation.test.mjs:257-261) does not mean SwiftPM's sandbox would apply; the SwiftPM-shaped profile still exits 71 under that outer profile. The refusal text and the record name say "SwiftPM manifest and plugin sandboxes", so a pass is a false positive for the incompatibility the PR detects. runnerIsolationPolicy always emits two deny rules, so dispatch cannot hit the pass today; the retained evidence and the docs are what are wrong.
• README.md:160-162 ("only a deny-free profile permits nesting") and docs/agent-benchmark.md:111-113 state a kernel rule that is incomplete in a way that matters for the #469 decision: no sandbox-exec outer profile at all can host SwiftPM's sandbox. Option 2 in the issue write-up ("Drop every deny rule, since a deny-free profile permits nesting") has a false premise; it is rejected there for another reason, but the maintainer is choosing from that list, and the corrected fact is that the trade-off is not isolation vs nesting. The write-up needs the same correction.

Suggested fix: probe with a deny-containing inner profile that mirrors SwiftPM's (the SwiftPM-shaped one above runs /usr/bin/true standalone with exit 0), correct the two doc passages and the issue comment, and change the darwin test's positive control to the same inner command run unsandboxed (plain execFileSync), which is the only control that exists on this kernel and still catches a broken invocation.

2. Refusal evidence is not retained, and the refusal lands after the expensive setup (driver.mjs:1367-1371, docs/agent-benchmark.md:117)

On refusal the function throws before the assignment, so preflight.isolationCompatibility is never written anywhere and the only trace is the error on stderr; the docs sentence "the probe results are retained with the private run metadata" holds only for runs that pass. By that point results/<stage>/<runId> already holds the copied golden control-tmp, the copied golden stim-home (stim arm), runner-home, prompt.txt, devices-before.json, runner-isolation.sb and, for launch-crash, the fixture checkout, with no meta.json or run.json (report() filters on run.json, so it ignores the directory; collect() would throw on it). That shape is pre-existing for any throw after the run directory is created, but this PR adds a refusal that is deterministic on this kernel for every iOS dispatch without the adapter, after waitForLoadGate (up to 10 minutes) and the golden copies. Non-blocking; at minimum attach the record to the error or write it into the run directory before throwing, and drop the "retained" wording for refused runs.

3. smoke <arm> does not run the probes (driver.mjs:1097-1134)

smoke has no platform parameter, so it cannot apply the iOS refusal, and dispatch does gate every timed run before the clock, which is what #469 asks for. The gap is that an operator cannot see the two probe results without dispatching; recording them in smoke.json would close it. Non-blocking.

Verified

• Claim 1 (outer deny rules): (allow default) alone nests /usr/bin/true with exit 0; plus a single deny of file-read-data (canonical literal, with a control showing the deny blocks a direct cat), mach-lookup, or sysctl-read -> 71 sandbox-exec: sandbox_apply: Operation not permitted; a deny plus (allow process-exec* (with no-sandbox) (literal "/usr/bin/sandbox-exec")) -> nested exit 0 and the nested cat read the protected file while a direct cat under the same profile stayed denied. Three deny-free levels nest fine.
• Claim 2 (ps): /bin/ps is -rwsr-xr-x root wheel; sandbox-exec -p '(version 1)(allow default)' /bin/ps -p $$ -o lstart= -> execvp() of '/bin/ps' failed: Operation not permitted, exit 71; the PR's exact /bin/sh -c '/bin/ps -p "$$" -o lstart=' -> /bin/sh: /bin/ps: Operation not permitted, exit 126. agent-device 0.20.10 (global install, version checked) dist/src/host-process.js sets i = '/bin/ps' and d(pid, field) runs /bin/ps -p <pid> -o lstart=|command=; platform-runtime-screen-recording-apple-simulator-host.js polls it 81 times before throwing simctl recordVideo did not expose a complete process identity. Importing the unpatched c/s/i/a exports under (allow default) and under a runnerIsolationPolicy-generated profile gives {"start":null,"command":null,"zombie":false,"listError":"spawn EPERM"}; unsandboxed they return the real start time, command and 945 processes.
• Claim 3 (wiring): both probes go through isolatedRunnerInvocation (digest check, launcher -p <policy> ...); the driver's run() is execFileSync with piped stderr and encoding: 'utf8', so a non-zero exit throws with .stderr; the call sits after prepareRunIsolation and before probeNativeCompatibility, verifyRunnerProfile, prepareAgentDeviceRun, dispatchAt and both spawns, so a refusal precedes any timing; the record reaches meta.preflight on success.
• Exemptions: verifyNativeCompatibility returns null unless STIM_BENCH_NATIVE_COMPAT_MANIFEST is set and hash-verified (and throws when a pin expects one), so the exemption needs a verified adapter. The adapter covers both mechanisms: ps via the patched host-process.js -> ps-bridge.mjs -> compiled non-setuid native-process, which probeNativeCompatibility asserts under the same isolated invocation; nesting via the compat/xcodebuild wrapper symlinked into allowed-bin ahead of /usr/bin on the runner PATH plus the patched build-xcframework.sh. Stim runs xcodebuild PATH-resolved (packages/stim-cli/src/engine/xcode.ts:680). The wrapper only covers PATH-resolved xcodebuild, so an xcrun xcodebuild, absolute-path or user-script-sandbox nesting would still fail inside the timed window, but that fails the build and the existing app/device proof marks such a run invalid; it cannot yield a valid-looking timing. Android: Gradle applies no sandbox and agent-device's Android recording identity reads /proc/<pid>/stat and /proc/<pid>/cmdline over adb shell, not host ps.
• Tests: the decision test has no darwin dependency (verifiedIsolation writes a profile; isolatedRunnerInvocation only hashes it), the darwin test is skipIf, the unit CI job is ubuntu-latest, fixture roots are removed in afterEach, and real sandbox-exec in the suite matches the three pre-existing darwin tests.
• Commands: pnpm exec vitest run scripts/agent-benchmark/runner-isolation.test.mjs 7/7; pnpm run lint, pnpm run format:check, pnpm run knip, pnpm run build, pnpm run typecheck clean; pnpm test after build: 3704 passed, 2 failed in engine-ios-device.test.ts from stale stim-devicectl-* entries in TMPDIR, matching the description.
• Not exercised by me either: real simctl recordVideo, real xcodebuild package resolution, the XCFramework build, dispatch end to end, and where the agent-device daemon that performs the recording is spawned relative to the sandbox (the issue's observed failure is the evidence that it runs inside it).

[janicduplessis]

Addressed the review in f4951d8.

1. Nested-sandbox probe: fixed. Reproduced the matrix and extended it. The rule on this kernel is narrower than the PR stated: a nested sandbox_apply succeeds only for a profile equivalent to the one already applied (identical, reordered, duplicated or redundant rules all nest, at any depth, deny rules or not), and any profile that differs in effect is refused in both directions. For SwiftPM the conclusion is the review's: its profile is per-invocation and (deny default)-based, so no runner policy hosts it. The probe now applies (version 1)(deny default)(import "system.sb")(allow file-read*)(allow process*), the head of SwiftPM's profile; the darwin test's positive control runs that inner command unsandboxed; the injected-execute test asserts the emitted inner profile is deny-default based; README and docs state the measured rule; the issue write-up is corrected in Benchmark iOS isolation needs native toolchain and recording compatibility checks #469 (comment).
2. Refusal evidence: fixed; order unchanged. The refusal error carries the probe record and dispatch writes it to isolation-compatibility.json in the run directory before rethrowing; the docs say where a refused run's results live. The probe stays after the golden copies because the policy grants the fixture checkout, guidance file, STIM_HOME and tmp paths that this setup creates, so moving it earlier would probe a different policy.
3. smoke <arm>: left as is, per the review; dispatch gates every timed run.

[janicduplessis]

Verdict: CLEAR (second-round review of f4951d8; no blocking finding, one non-blocking doc correction. Posted as a comment review because GitHub refuses approve and request-changes from the author account.)

Fresh review of the diff vs main, the first review, the fix-up, #469 and its correction comment, with the sandbox claims re-measured on macOS 26.5 (25F71).

Findings

1. The README states the nesting rule as "equivalent in effect"; the kernel is stricter than that (non-blocking) — scripts/agent-benchmark/README.md:160-164

The paragraph says a nested sandbox_apply succeeds for a profile "equivalent to the one the process already runs under" and that "a profile that differs in effect is refused". The second half held in every row I ran; the first does not. A profile that differs only by a rule with no effect is refused too:

outer (allow default) / inner (allow default)(deny file-read-data (literal "/etc/hosts"))         -> 71 sandbox_apply: Operation not permitted
outer (allow default)(deny file-read-data (literal "/etc/hosts")) / inner (allow default)         -> 71
outer (allow default) / inner (allow default)(deny file-read-data (literal "/nonexistent/never"))  -> 71

Both denies are dead: the sandbox matches the canonical /private/etc/hosts, so a direct cat /etc/hosts under either profile exits 0 (with (literal "/private/etc/hosts") it exits 1). The two profiles in each row are identical in effect and still refused, while reordered, duplicated, redundant-allow, (version 2) and (regex #"^/private/etc/hosts$") variants of a live deny all nest (exit 0). What nests is a profile that compiles to the same sandbox as the applied one, not one equivalent in effect. The operative conclusion is unchanged and if anything firmer: SwiftPM's per-invocation profile can never match a runner policy. docs/agent-benchmark.md:111-113 ("any profile that differs from the one a process already runs under") already says it correctly; the README sentence, the PR body's measurements preamble and the #469 correction comment carry the "equivalent"/"differs in effect" wording. Suggested README wording: "unless the inner profile compiles to the same sandbox the process already runs under (identical, reordered, duplicated or redundant rules all nest); any other profile is refused in both directions, even one that only adds a rule with no effect."

First-round findings

• Finding 1 (deny-free inner profile): resolved. runner-isolation.mjs:155-161 nests (version 1)(deny default)(import "system.sb")(allow file-read*)(allow process*) through isolatedRunnerInvocation, so the inner sandbox-exec runs under the exact verified policy. Those five fragments are, in that order, the first five profile strings after /usr/bin/sandbox-exec in Xcode's Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-package (followed by the two mach-lookup grants, the cache-write regex, the network block, file-write* and sysctl*), and match macOSSandboxProfile in Sandbox.swift at the linked commit. On this kernel permitted: true can only arise when the runner policy compiles to the same sandbox as that head, which no policy with a write grant can, so the probe cannot report a false pass under any runnerIsolationPolicy output; the darwin test pins the refusal.
• Finding 2 (evidence retention): resolved. The refusal error carries isolationCompatibility; driver.mjs:1367-1381 writes isolation-compatibility.json into runDir from the driver process (no sandbox-exec involved) and rethrows. Only a refusal writes it; a passing record reaches meta.preflight. Nothing lists a run directory expecting a fixed file set (report filters on run.json, collect reads meta.json/run.json, the only run-directory listing is proof/). The probe order stays after the golden copies: the policy's scoped grants come from crash.fixtureCheckout, claudeGuidance.path, env.STIM_HOME and runDir/tmp, and prepareRunnerIsolation needs runDir/tmp to exist, so an earlier probe would verify a different policy.
• Finding 3 (smoke): left as is, non-blocking, as agreed.

Verified

• Nesting matrix, /usr/bin/sandbox-exec -p <outer> /usr/bin/sandbox-exec -p <inner> /usr/bin/true, swiftpm = the probe's inner profile: (allow default)/(allow default) 0, with a redundant (allow process-exec*) 0; (allow default)/swiftpm 71; (allow default)/(allow default)+live deny 71; live deny/itself 0, reordered 0, duplicated 0, plus redundant allow 0, regex form 0, (version 2) 0; deny X/deny Y 71; deny X/deny X+Y 71; deny X+Y/deny X 71; deny X/(allow default) 71; swiftpm/swiftpm 0 and three levels deep 0; swiftpm/swiftpm+(allow file-write* (subpath "/private/tmp")) 71 and the reverse 71; swiftpm/(allow default) 71; swiftpm and swiftpm+write standalone 0. Exact runnerIsolationPolicy output, generated through the module with temporary roots: /swiftpm 71, /(allow default) 71, /itself 0, /bin/sh -c '/bin/ps -p "$$" -o lstart=' 126 /bin/sh: /bin/ps: Operation not permitted, standalone /usr/bin/true 0. Escape row with a live deny: direct cat exit 1, nested (allow default) cat under (allow process-exec* (with no-sandbox) (literal "/usr/bin/sandbox-exec")) read the file.
• ps: /bin/ps is -rwsr-xr-x root wheel; the probe command exits 126 under (allow default) and under the exact policy, 0 unsandboxed. Unpatched agent-device 0.20.10 host-process.js exports (c/s/i/a) under (allow default): {"start":null,"command":null,"zombie":false,"listError":"spawn EPERM"}; unsandboxed: real start time, command, 948 processes. Android recording identity reads /proc/<pid>/stat and /proc/<pid>/cmdline (platform-runtime-screen-recording-android-host.js), so the Android exemption holds. The adapter exemption still needs verifyNativeCompatibility (null without STIM_BENCH_NATIVE_COMPAT_MANIFEST, throws when the pin expects one), and probeNativeCompatibility then asserts the patched identity path and the xcodebuild wrapper under the same isolated invocation, so it cannot admit an unproven timing.
• Tests: pnpm exec vitest run scripts/agent-benchmark/runner-isolation.test.mjs 7/7 on darwin with real sandbox-exec, no stim-runner-isolation-* left in TMPDIR; the injected-execute test has no darwin dependency and the darwin test is skipIf. Two throwaway mutants of the unit test (isolationCompatibility: { bogus: true }, and a wrong stringContaining message) each fail with "expected error to match asymmetric matcher", so the toThrow(expect.objectContaining(...)) assertion is load-bearing under vitest 5. The darwin positive control runs /usr/bin/sandbox-exec -p <swiftpm> /usr/bin/true unsandboxed and asserts no throw; the nested run asserts the exact sandbox_apply stderr from real output.
• pnpm run lint, format:check, knip, build, typecheck clean; pnpm test: 3704 passed, 1 skipped, 2 failed in engine-ios-device.test.ts (stale stim-devicectl-* in TMPDIR, pre-existing, as the description says). gh pr checks 551: test (node 22), test (node 24) and published runtime pass; device jobs skipped.
• Docs and issue: README and docs/agent-benchmark.md name the two mechanisms and where refused-run results live, and the existing sentence that this does not replace a real untimed build and recording test stands; the #469 correction identifies option 2's false premise correctly. The one added code comment names the external source (SwiftPM's Sandbox.swift) and the constraint, which fits the comment policy. The PR body describes the final diff without review-round narration.
• Not exercised by me either: bench.mjs dispatch end to end, a real simctl recordVideo, real xcodebuild package resolution or the XCFramework build; I booted no simulator.

[janicduplessis]

Second-round follow-up: the README now states the rule as "compiles to the same sandbox", with the dead-deny case called out; the description's measurement preamble uses the same wording, and a precision note is posted on #469. Docs-only commit, no code change.