Skip to content

test(e2e): pin the contracts a README walkthrough expects (EAI-8024) - #241

Open
fredespi wants to merge 8 commits into
mainfrom
e2e-pin-the-contracts-for-the-13-readme
Open

test(e2e): pin the contracts a README walkthrough expects (EAI-8024)#241
fredespi wants to merge 8 commits into
mainfrom
e2e-pin-the-contracts-for-the-13-readme

Conversation

@fredespi

Copy link
Copy Markdown
Collaborator

Walking the public README end to end on an MI300X pod turned up thirteen places
where the CLI does not do what the README leads a reader to expect. This adds
end-to-end coverage for them. It is test-only — no product behaviour changes.

Each scenario asserts the behaviour the README promises and is registered as an
expected failure in expectations.toml, citing its ticket. CI stays green, and
the day a defect is fixed its row becomes a stale XPASS and deletes itself.

Pinned as expected failures

Ticket What the scenario pins Lane
EAI-8011 The help offers rocm serve qwen2.5-7b-instruct, which is neither a name the model listing knows nor an owner/repo reference no-GPU
EAI-8011 The help describes running rocm with no subcommand as opening the dashboard — what it also says rocm dash does no-GPU
EAI-8010 rocm update --dry-run is refused as a usage error, though the README documents --apply and --dry-run as independent no-GPU
EAI-8019 serve --help advertises cpu_only and the command refuses it for being cpu_only no-GPU
EAI-8020 The diagnosis and the fix preview print different verification steps for the same remedy no-GPU
EAI-8009 Chatting locally without naming a model ignores a ready local server that is not the built-in assistant model no-GPU
EAI-8014 Uninstall reports completion while leaving the servers it manages running, having deleted the records needed to stop them no-GPU
EAI-8015 A remedy tells the user to join UNKNOWN — the group-lookup failure, not a group MI300X
EAI-8020 The diagnosis and the fix propose adding the user to different groups MI300X
EAI-8021 A device the user can already read and write is still offered as a group-permission cause MI300X
EAI-8016 A second server is given the default address even when it is already in use MI300X

Two ship as guards, with no expected-failure row

EAI-8007 (a stop that worked reporting it stopped nothing) and EAI-8006
(--gpu auto selecting none) were reproduced on the pod but hold correctly on
every lane available here — EAI-8007 against both a plain registered process and
a real vLLM serve, EAI-8006 because the runner's amd-smi executes where the
pod's bundled one could not. A row would be a stale XPASS and would fail the
lane, so their scenarios ship as guards for the contract instead. Both tickets
keep the pod observation.

Two are not covered, deliberately

EAI-8017 and EAI-8018 both need a working Lemonade install, and Lemonade's
launcher cannot start on either lane: on the MI300X runner it wants GLIBC_2.38
and GLIBCXX_3.4.32, which that OS does not have, and in a plain container it
wants libdrm_amdgpu.so.1. Each dies before reaching the behaviour under test,
so scenarios for them passed while proving nothing. They were removed rather
than shipped as false coverage, and stay on their tickets.

Harness additions

Confined to the test crate: removing environment variables from the child (the
existing helper only sets them), stat and id shims scoped to one device path
so a device-group state can be arranged rather than hoped for, and a child
process a scenario owns and registers as a managed service so it can ask whether
the product stopped it — asserted before teardown, which would otherwise clean up
after the product and hide the defect.

Two caveats worth stating. A substituted stat answer proves how the CLI handles
an unnameable group, not that a given host produces one; the provenance for that
is the pod. And the device-group scenarios need a real device node, so they run
on the GPU lane only.

Test plan

  • No-GPU lane, three independent runs: 52 scenarios, 10 xfail, 0 XPASS, 0
    unexpected failures.
  • Full MI300X lane (e2e-selfhosted, no name filter, so expectations resolve
    normally): 64 scenarios, 16 xfail, 0 XPASS, 0 unexpected failures.
  • Every new scenario was read individually to confirm it fails for the reason its
    row states, rather than because a step is miswired.
  • cargo clippy --workspace --all-targets -D warnings, the workspace tests, and
    cargo test -p e2e-cucumber --lib pass in a Linux container.

Four rows were corrected from what the GPU dispatches measured rather than left
as first written, which is why two became guards and two scenarios were dropped.

@fredespi
fredespi requested a review from a team as a code owner August 12, 2026 19:09
@fredespi
fredespi force-pushed the e2e-pin-the-contracts-for-the-13-readme branch 3 times, most recently from d94221b to 9fc2de0 Compare August 13, 2026 07:18
fredespi and others added 7 commits August 13, 2026 14:47
Walking the public README end to end on an MI300X pod turned up thirteen
places where the CLI does not do what the README leads a reader to expect.
This adds a scenario for each and registers it as an expected failure, so
CI stays green and every row goes stale the day its defect is fixed.

No product behaviour changes.

Pinned on the no-GPU lane, so they gate every PR:
  EAI-8011  the help offers `rocm serve qwen2.5-7b-instruct`, which is
            neither a name the model listing knows nor an owner/repo
            reference
  EAI-8011  the help describes running `rocm` with no subcommand as
            opening the dashboard, which is what it says `rocm dash` does
  EAI-8010  `rocm update --dry-run` is refused as a usage error, though
            the README documents --apply and --dry-run as independent
  EAI-8019  `serve --help` advertises cpu_only and the command refuses it
  EAI-8020  the diagnosis and the fix preview print different verification
            steps for the same remedy
  EAI-8009  chatting locally with no --model ignores a ready local server
            that is not the built-in assistant model
  EAI-8014  uninstall reports completion while leaving the servers it
            manages running, having deleted the records needed to stop them

Pinned on the GPU lane, scoped `when = {}` for a first measurement:
  EAI-8007  a stop that worked reports that it stopped no processes
  EAI-8015  a remedy names the group-lookup failure UNKNOWN as a group
  EAI-8020  the diagnosis and the fix propose different group commands
  EAI-8021  a device the user can already read and write is still offered
            as a group-permission cause
  EAI-8006  `--gpu auto` selects none on a machine that has a GPU
  EAI-8016  a second server is given an address already in use
  EAI-8018  Lemonade is advertised ready on the GPU but never answers
  EAI-8017  installing an engine fails when the session provides no
            scratch directory

Harness additions, all small: removing environment variables from the
child (not just setting them), a `stat` shim scoped to one device path so
a device-group state can be arranged rather than hoped for, and a child
process a scenario owns and registers as a managed service so it can ask
whether the product stopped it — asserted before teardown, which would
otherwise clean up after the product and hide the defect.

EAI-8007 was first written against that child-process fixture on the
no-GPU lane and XPASSed: a record with no start-time token takes the
legacy best-effort path and is counted correctly. It now serves a real
model instead, which is the path the pod exercised.

Verified in a Linux container: ten new scenarios fail, each for the reason
its row states, with no XPASS and no new unexpected failures. The one
remaining failure there, engine-shell-marks-the-prompt, reproduces
identically on unmodified main — the container runs as root, so the prompt
ends in `#` where the assertion looks for `$`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
The first app-dev-gpu dispatch answered the MEASURE questions the rows were
carrying, and three of them came back differently from the plan.

EAI-8007 loses its row. Neither fixture reproduces it: a plain process
registered as a managed service is counted correctly (no-GPU lane), and so is
a real vLLM serve (MI300X). A row would be a stale XPASS on both lanes and
would fail them, so the scenario ships as a guard for the contract the pod
violated instead. What was different about that host is still unknown.

EAI-8021 needed a second substituted answer. The scenario made the device
world-writable and expected the group-permission cause to stop being offered,
but the GPU runner's user is already in the conventional groups, so the cause
scored zero there for a reason that has nothing to do with device access and
the scenario passed. It now substitutes the group answer too, so the state it
describes — access works, membership absent — holds on any host.

EAI-8006 was failing for the wrong reason: the serve was refused for having no
active ROCm runtime, before it ever chose a device. It now takes the same
managed-runtime precondition its sibling serve scenarios do.

Two scenarios could not be measured at all, because `rocm engines install
lemonade` fails on the GPU runner within two seconds:
  - EAI-8017 was passing vacuously — the install never got far enough to leave
    the engine without a scratch directory, and the assertion tolerated that.
    It now requires the install to have got under way before it will speak.
  - EAI-8018 asserted a claim the inventory never makes there, because
    Lemonade is not installed in the scenario's isolated directories. It now
    installs it first and reports the install output when the claim is absent.
Both will say why on the next dispatch.

Also scopes the diagnosis comparisons to the cause under test rather than the
first one reported (which cause ranks highest is host-dependent), tightens the
scratch-directory match so ordinary path output cannot satisfy it, and stops
the uninstall assertion passing vacuously when no server was registered.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second dispatch settled the remaining MEASURE questions.

EAI-8021 now fails for its stated reason once the group answer is substituted
alongside the device one (score 45 with the device readable and writable), so
its row stands.

EAI-8006 loses its row and ships as a guard. `--gpu auto` reported selecting
none on the pod, where the bundled amd-smi could not execute; the MI300X
runner's amd-smi runs, so the CLI names a device and the scenario passes there.

EAI-8017 and EAI-8018 lose their scenarios entirely. Both need a working
Lemonade install, and Lemonade's launcher cannot start on either lane: on the
MI300X runner it wants GLIBC_2.38 and GLIBCXX_3.4.32, which that OS does not
have, and in a plain container it wants libdrm_amdgpu.so.1. Each dies before
reaching the behaviour under test, so the scenarios were passing vacuously —
which reads as coverage while proving nothing. Better no scenario than that
one; both stay on their tickets, reproduced by hand on the pod, until a lane
can run Lemonade. Removing the Lemonade install from the engine-inventory
scenario also stops the suite writing into the runner's shared pre-warm tree,
which it had begun doing.

That leaves nine of the thirteen pinned as expected failures, two shipping as
guards for a contract our lanes uphold and the pod did not, and two not
covered.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#242 landed `Scenario: 14` in model_serving.feature while this branch was also
using 14, and the two touch different parts of the file — so git merges them
cleanly and the collision is invisible until someone tries to refer to
"scenario 14" and finds two of them.

Renumbers this branch's five additions to 15-19, after main's highest. No
scenario, tag, step or expectation changes.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Strix Halo Windows lane finally reported, and it contradicts the reading
this branch shipped with. `--gpu auto` really does report `selected none` on a
machine that has a GPU — just not on the lanes measured first.

  MI300X             gfx943  linux    passes
  Strix Halo Ubuntu  gfx1151 linux    passes
  Strix Halo Windows gfx1151 windows  fails: `gpu: auto (selected none)`

Same GPU family on both Strix lanes, so the discriminator is the OS rather than
the family — which is why this row uses `os = "windows"` and not the
`therock_family` scoping `examine-both-forms-agree-on-gpu` uses.

So the scenario stops being a bare guard and gets the expected-failure row it
should have had, scoped to where the defect was actually observed. It still
guards both Linux lanes, where the device is named.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… bug

`services-stop-reports-what-it-stopped` needs a real running server before it
can ask what stopping one reports, and that Given serves on whatever engine the
host defaults to. On a lemonade Linux host that is the serve which reaches ready
and is shut down ~0.08s later (EAI-7423), so the PRECONDITION fails and the
scenario never reaches the contract it exists to hold.

Measured on Strix Halo Ubuntu: passed in run 196, failed in run 210 with
"endpoint did not serve model Qwen3-0.6B after 2 attempt(s) of 300s" — the
EAI-7423 signature its sibling rows already describe.

Adds the row those siblings carry, scoped identically: os=linux because native
Windows lemonade serve works and this passes there, gfx* because it needs a real
GPU host, and flaky=true because which of these pass varies run to run and a
deterministic-XPASS row would fail that lane on unrelated PRs.

The row cites EAI-7423, not EAI-8007 — the stop-reporting contract itself still
holds everywhere it has been observed, so it keeps no expected-failure row of
its own.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Strix Halo lanes cap at 35 minutes, and on main alone the Ubuntu one already
runs 32.3 of them. This branch adds three scenarios that each start a real
server, costing about nine more minutes on that lane, so both Strix jobs being
cancelled at the cap is arithmetic rather than bad luck.

Tags them @merge-queue, the mechanism scenarios 6, 6b and 8 already use for
exactly this: they drop off the PR fast path and run in the merge queue, where
the budget for heavy serves lives and where all three self-hosted lanes still
execute them before anything lands. Measured on Strix Ubuntu after the
fail-fast row in the previous commit: 232s, 136s and 149s.

Everything this branch pins that does NOT need a running server keeps running on
every PR, on every lane, unchanged: the help contracts, update, device policies,
the diagnose comparisons and shims, local chat, and uninstall.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fredespi
fredespi force-pushed the e2e-pin-the-contracts-for-the-13-readme branch from 9fc2de0 to 39afe38 Compare August 13, 2026 12:49

@juhovainio juhovainio left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed this against gh pr diff 241 — good, well-documented test-only PR (no product code touched), and I cross-checked all 13 new scenarios against their expectations.toml rows and lane tags; that bookkeeping is consistent and the reasoning for each row (including the two tickets deliberately left uncovered because Lemonade can't start on either available lane) holds up.

One real bug in the new harness code, though: advertised_device_policies in serving_steps.rs reads the wrong flag's help text, which breaks the exact scenario it's meant to support. Left as an inline comment with a fix suggestion. Also left a smaller nit on find_bool in diagnose_steps.rs that's not blocking but worth a look.

Comment thread tests/e2e-cucumber/tests/e2e/serving_steps.rs
Comment thread tests/e2e-cucumber/tests/e2e/diagnose_steps.rs Outdated
The device-policy scenario scraped the FIRST '[possible values:' line out of
'serve --help', which is --engine's clap-generated list, not --device's. So it
served '--device lemonade' and '--device vllm', which are refused as unsupported
policies -- wording close enough to the cpu_only rejection that the assertion
matched and the scenario looked like it was failing for the reason its row
states. It was not, and it could never have gone stale: those two values stay
invalid whatever happens to EAI-8019, so a fixed bug would still have read as
unfixed.

Scopes the lookup to --device's own block, stopping at the next option so a
--device that advertises nothing cannot borrow a neighbour's list. Verified
against the real 'serve --help': the parser now yields gpu_required,
gpu_preferred, cpu_only.

Introspecting clap's device arg directly would be better still, but this suite
is black-box by design and cannot import the product crates; main.rs already
guards that list from the inside.

Also reads user_can_read/user_can_write from the kfd device by name rather than
searching the whole report. Every device in it carries those fields, so the
generic walk was answered by whichever sorted first rather than by the device
the scenario shimmed.

Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@fredespi
fredespi requested a review from juhovainio August 13, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants