test(e2e): pin the contracts a README walkthrough expects (EAI-8024) - #241
Open
fredespi wants to merge 8 commits into
Open
test(e2e): pin the contracts a README walkthrough expects (EAI-8024)#241fredespi wants to merge 8 commits into
fredespi wants to merge 8 commits into
Conversation
fredespi
force-pushed
the
e2e-pin-the-contracts-for-the-13-readme
branch
3 times, most recently
from
August 13, 2026 07:18
d94221b to
9fc2de0
Compare
Walking the public README end to end on an MI300X pod turned up thirteen
places where the CLI does not do what the README leads a reader to expect.
This adds a scenario for each and registers it as an expected failure, so
CI stays green and every row goes stale the day its defect is fixed.
No product behaviour changes.
Pinned on the no-GPU lane, so they gate every PR:
EAI-8011 the help offers `rocm serve qwen2.5-7b-instruct`, which is
neither a name the model listing knows nor an owner/repo
reference
EAI-8011 the help describes running `rocm` with no subcommand as
opening the dashboard, which is what it says `rocm dash` does
EAI-8010 `rocm update --dry-run` is refused as a usage error, though
the README documents --apply and --dry-run as independent
EAI-8019 `serve --help` advertises cpu_only and the command refuses it
EAI-8020 the diagnosis and the fix preview print different verification
steps for the same remedy
EAI-8009 chatting locally with no --model ignores a ready local server
that is not the built-in assistant model
EAI-8014 uninstall reports completion while leaving the servers it
manages running, having deleted the records needed to stop them
Pinned on the GPU lane, scoped `when = {}` for a first measurement:
EAI-8007 a stop that worked reports that it stopped no processes
EAI-8015 a remedy names the group-lookup failure UNKNOWN as a group
EAI-8020 the diagnosis and the fix propose different group commands
EAI-8021 a device the user can already read and write is still offered
as a group-permission cause
EAI-8006 `--gpu auto` selects none on a machine that has a GPU
EAI-8016 a second server is given an address already in use
EAI-8018 Lemonade is advertised ready on the GPU but never answers
EAI-8017 installing an engine fails when the session provides no
scratch directory
Harness additions, all small: removing environment variables from the
child (not just setting them), a `stat` shim scoped to one device path so
a device-group state can be arranged rather than hoped for, and a child
process a scenario owns and registers as a managed service so it can ask
whether the product stopped it — asserted before teardown, which would
otherwise clean up after the product and hide the defect.
EAI-8007 was first written against that child-process fixture on the
no-GPU lane and XPASSed: a record with no start-time token takes the
legacy best-effort path and is counted correctly. It now serves a real
model instead, which is the path the pod exercised.
Verified in a Linux container: ten new scenarios fail, each for the reason
its row states, with no XPASS and no new unexpected failures. The one
remaining failure there, engine-shell-marks-the-prompt, reproduces
identically on unmodified main — the container runs as root, so the prompt
ends in `#` where the assertion looks for `$`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The first app-dev-gpu dispatch answered the MEASURE questions the rows were
carrying, and three of them came back differently from the plan.
EAI-8007 loses its row. Neither fixture reproduces it: a plain process
registered as a managed service is counted correctly (no-GPU lane), and so is
a real vLLM serve (MI300X). A row would be a stale XPASS on both lanes and
would fail them, so the scenario ships as a guard for the contract the pod
violated instead. What was different about that host is still unknown.
EAI-8021 needed a second substituted answer. The scenario made the device
world-writable and expected the group-permission cause to stop being offered,
but the GPU runner's user is already in the conventional groups, so the cause
scored zero there for a reason that has nothing to do with device access and
the scenario passed. It now substitutes the group answer too, so the state it
describes — access works, membership absent — holds on any host.
EAI-8006 was failing for the wrong reason: the serve was refused for having no
active ROCm runtime, before it ever chose a device. It now takes the same
managed-runtime precondition its sibling serve scenarios do.
Two scenarios could not be measured at all, because `rocm engines install
lemonade` fails on the GPU runner within two seconds:
- EAI-8017 was passing vacuously — the install never got far enough to leave
the engine without a scratch directory, and the assertion tolerated that.
It now requires the install to have got under way before it will speak.
- EAI-8018 asserted a claim the inventory never makes there, because
Lemonade is not installed in the scenario's isolated directories. It now
installs it first and reports the install output when the claim is absent.
Both will say why on the next dispatch.
Also scopes the diagnosis comparisons to the cause under test rather than the
first one reported (which cause ranks highest is host-dependent), tightens the
scratch-directory match so ordinary path output cannot satisfy it, and stops
the uninstall assertion passing vacuously when no server was registered.
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second dispatch settled the remaining MEASURE questions. EAI-8021 now fails for its stated reason once the group answer is substituted alongside the device one (score 45 with the device readable and writable), so its row stands. EAI-8006 loses its row and ships as a guard. `--gpu auto` reported selecting none on the pod, where the bundled amd-smi could not execute; the MI300X runner's amd-smi runs, so the CLI names a device and the scenario passes there. EAI-8017 and EAI-8018 lose their scenarios entirely. Both need a working Lemonade install, and Lemonade's launcher cannot start on either lane: on the MI300X runner it wants GLIBC_2.38 and GLIBCXX_3.4.32, which that OS does not have, and in a plain container it wants libdrm_amdgpu.so.1. Each dies before reaching the behaviour under test, so the scenarios were passing vacuously — which reads as coverage while proving nothing. Better no scenario than that one; both stay on their tickets, reproduced by hand on the pod, until a lane can run Lemonade. Removing the Lemonade install from the engine-inventory scenario also stops the suite writing into the runner's shared pre-warm tree, which it had begun doing. That leaves nine of the thirteen pinned as expected failures, two shipping as guards for a contract our lanes uphold and the pod did not, and two not covered. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#242 landed `Scenario: 14` in model_serving.feature while this branch was also using 14, and the two touch different parts of the file — so git merges them cleanly and the collision is invisible until someone tries to refer to "scenario 14" and finds two of them. Renumbers this branch's five additions to 15-19, after main's highest. No scenario, tag, step or expectation changes. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Strix Halo Windows lane finally reported, and it contradicts the reading this branch shipped with. `--gpu auto` really does report `selected none` on a machine that has a GPU — just not on the lanes measured first. MI300X gfx943 linux passes Strix Halo Ubuntu gfx1151 linux passes Strix Halo Windows gfx1151 windows fails: `gpu: auto (selected none)` Same GPU family on both Strix lanes, so the discriminator is the OS rather than the family — which is why this row uses `os = "windows"` and not the `therock_family` scoping `examine-both-forms-agree-on-gpu` uses. So the scenario stops being a bare guard and gets the expected-failure row it should have had, scoped to where the defect was actually observed. It still guards both Linux lanes, where the device is named. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… bug `services-stop-reports-what-it-stopped` needs a real running server before it can ask what stopping one reports, and that Given serves on whatever engine the host defaults to. On a lemonade Linux host that is the serve which reaches ready and is shut down ~0.08s later (EAI-7423), so the PRECONDITION fails and the scenario never reaches the contract it exists to hold. Measured on Strix Halo Ubuntu: passed in run 196, failed in run 210 with "endpoint did not serve model Qwen3-0.6B after 2 attempt(s) of 300s" — the EAI-7423 signature its sibling rows already describe. Adds the row those siblings carry, scoped identically: os=linux because native Windows lemonade serve works and this passes there, gfx* because it needs a real GPU host, and flaky=true because which of these pass varies run to run and a deterministic-XPASS row would fail that lane on unrelated PRs. The row cites EAI-7423, not EAI-8007 — the stop-reporting contract itself still holds everywhere it has been observed, so it keeps no expected-failure row of its own. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Strix Halo lanes cap at 35 minutes, and on main alone the Ubuntu one already runs 32.3 of them. This branch adds three scenarios that each start a real server, costing about nine more minutes on that lane, so both Strix jobs being cancelled at the cap is arithmetic rather than bad luck. Tags them @merge-queue, the mechanism scenarios 6, 6b and 8 already use for exactly this: they drop off the PR fast path and run in the merge queue, where the budget for heavy serves lives and where all three self-hosted lanes still execute them before anything lands. Measured on Strix Ubuntu after the fail-fast row in the previous commit: 232s, 136s and 149s. Everything this branch pins that does NOT need a running server keeps running on every PR, on every lane, unchanged: the help contracts, update, device policies, the diagnose comparisons and shims, local chat, and uninstall. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fredespi
force-pushed
the
e2e-pin-the-contracts-for-the-13-readme
branch
from
August 13, 2026 12:49
9fc2de0 to
39afe38
Compare
juhovainio
requested changes
Aug 13, 2026
juhovainio
left a comment
Collaborator
There was a problem hiding this comment.
Reviewed this against gh pr diff 241 — good, well-documented test-only PR (no product code touched), and I cross-checked all 13 new scenarios against their expectations.toml rows and lane tags; that bookkeeping is consistent and the reasoning for each row (including the two tickets deliberately left uncovered because Lemonade can't start on either available lane) holds up.
One real bug in the new harness code, though: advertised_device_policies in serving_steps.rs reads the wrong flag's help text, which breaks the exact scenario it's meant to support. Left as an inline comment with a fix suggestion. Also left a smaller nit on find_bool in diagnose_steps.rs that's not blocking but worth a look.
The device-policy scenario scraped the FIRST '[possible values:' line out of 'serve --help', which is --engine's clap-generated list, not --device's. So it served '--device lemonade' and '--device vllm', which are refused as unsupported policies -- wording close enough to the cpu_only rejection that the assertion matched and the scenario looked like it was failing for the reason its row states. It was not, and it could never have gone stale: those two values stay invalid whatever happens to EAI-8019, so a fixed bug would still have read as unfixed. Scopes the lookup to --device's own block, stopping at the next option so a --device that advertises nothing cannot borrow a neighbour's list. Verified against the real 'serve --help': the parser now yields gpu_required, gpu_preferred, cpu_only. Introspecting clap's device arg directly would be better still, but this suite is black-box by design and cannot import the product crates; main.rs already guards that list from the inside. Also reads user_can_read/user_can_write from the kfd device by name rather than searching the whole report. Every device in it carries those fields, so the generic walk was answered by whichever sorted first rather than by the device the scenario shimmed. Signed-off-by: fredespi <fredrik.espinoza@gmail.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Walking the public README end to end on an MI300X pod turned up thirteen places
where the CLI does not do what the README leads a reader to expect. This adds
end-to-end coverage for them. It is test-only — no product behaviour changes.
Each scenario asserts the behaviour the README promises and is registered as an
expected failure in
expectations.toml, citing its ticket. CI stays green, andthe day a defect is fixed its row becomes a stale XPASS and deletes itself.
Pinned as expected failures
rocm serve qwen2.5-7b-instruct, which is neither a name the model listing knows nor anowner/reporeferencerocmwith no subcommand as opening the dashboard — what it also saysrocm dashdoesrocm update --dry-runis refused as a usage error, though the README documents--applyand--dry-runas independentserve --helpadvertisescpu_onlyand the command refuses it for beingcpu_onlyUNKNOWN— the group-lookup failure, not a groupTwo ship as guards, with no expected-failure row
EAI-8007 (a stop that worked reporting it stopped nothing) and EAI-8006
(
--gpu autoselecting none) were reproduced on the pod but hold correctly onevery lane available here — EAI-8007 against both a plain registered process and
a real vLLM serve, EAI-8006 because the runner's
amd-smiexecutes where thepod's bundled one could not. A row would be a stale XPASS and would fail the
lane, so their scenarios ship as guards for the contract instead. Both tickets
keep the pod observation.
Two are not covered, deliberately
EAI-8017 and EAI-8018 both need a working Lemonade install, and Lemonade's
launcher cannot start on either lane: on the MI300X runner it wants GLIBC_2.38
and GLIBCXX_3.4.32, which that OS does not have, and in a plain container it
wants
libdrm_amdgpu.so.1. Each dies before reaching the behaviour under test,so scenarios for them passed while proving nothing. They were removed rather
than shipped as false coverage, and stay on their tickets.
Harness additions
Confined to the test crate: removing environment variables from the child (the
existing helper only sets them),
statandidshims scoped to one device pathso a device-group state can be arranged rather than hoped for, and a child
process a scenario owns and registers as a managed service so it can ask whether
the product stopped it — asserted before teardown, which would otherwise clean up
after the product and hide the defect.
Two caveats worth stating. A substituted
statanswer proves how the CLI handlesan unnameable group, not that a given host produces one; the provenance for that
is the pod. And the device-group scenarios need a real device node, so they run
on the GPU lane only.
Test plan
unexpected failures.
e2e-selfhosted, no name filter, so expectations resolvenormally): 64 scenarios, 16 xfail, 0 XPASS, 0 unexpected failures.
row states, rather than because a step is miswired.
cargo clippy --workspace --all-targets -D warnings, the workspace tests, andcargo test -p e2e-cucumber --libpass in a Linux container.Four rows were corrected from what the GPU dispatches measured rather than left
as first written, which is why two became guards and two scenarios were dropped.