Skip to content

test(e2e): run the E2E suite on the self-hosted WSL2 runner - #141

Open
rominf wants to merge 6 commits into
mainfrom
worktree-test-hosted-wsl-e2e
Open

test(e2e): run the E2E suite on the self-hosted WSL2 runner#141
rominf wants to merge 6 commits into
mainfrom
worktree-test-hosted-wsl-e2e

Conversation

@rominf

@rominf rominf commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • run the full black-box Cucumber suite in the Ubuntu-under-WSL2 distro on the self-hosted Strix Halo box (runs-on: [self-hosted, linux, strix-halo, wsl])
  • mirror the native self-hosted Linux lane step for step: stray-serve reclaim, bounded GPU preflight, toolchain bootstrap, shared-runtime pre-warm, then the whole suite with no hand filtering
  • add @requires-wsl / @requires-no-wsl capability tags, a WSL system-inspection scenario, and a distinct WSL platform identity in the report grid
  • key @requires-gpu on the driver verdict rather than the reported gfx target, so a WSL host without ROCm passthrough reports its GPU scenarios as not applicable instead of failing them

Why: WSL-specific behavior had parser and self-test coverage but nothing running inside a real WSL2 host. Nothing is filtered by hand — the capability probe resolves each scenario against the live host.

GPU preflight is advisory on this lane specifically: GPU access under WSL is what the lane is proving out, so a missing rocm-smi warns and continues. A GPU that is present but still held by a leftover serve still fails, since that would corrupt the serve scenarios.

Capability gating had to be fixed to make the lane honest

has_amd_gpu was gfx_target.is_some(). Under WSL the gfx target is read from the Windows-side driver and is reported whether or not ROCm can reach the device, so this runner advertised gfx1151 while rocm serve refused with "no usable AMD GPU".

Every @requires-gpu scenario therefore ran on a host that cannot serve one and failed on its premise — which is precisely what the capability probe exists to prevent. It surfaced as serve-absent-gpu-index-rejected, whose own comment already explained the trap ("on a no-GPU host the GPU-required pre-flight refuses before the index is ever validated") and which already carried @requires-gpu; the gate simply never fired. Eight further scenarios failed the same way and were only absorbed because unrelated platform-independent xfail entries happened to cover them.

The probe now takes the CLI's own verdict: on WSL it requires driver_status: wsl_rocdxg_ready, which the product sets only once /dev/dxg, dxcore, librocdxg.so and its ldconfig entry are all present. Native hosts are unaffected. On this runner the probe went from gpu=true to gpu=false, the nine premise failures became not-applicable, and the xfail count dropped from eight to three real ones.

What this lane covers today

The runner's WSL distro has /dev/dxg and dxcore but not librocdxg.so, so rocm examine reports driver_status: wsl_rocdxg_missing and the runtime cannot reach the GPU. The lane therefore currently exercises WSL host detection, the Windows-to-WSL execution boundary, and the whole non-GPU suite, with the GPU scenarios correctly reported as not applicable rather than failed.

That is a runner-configuration gap, not a gap in this change: once the passthrough is installed, driver_status flips to wsl_rocdxg_ready, the probe reports gpu=true, and the GPU scenarios begin running on this lane with no edit to this PR. docs/ci-hardware-testing.md documents what the distro needs for each tier of coverage.

Risk: medium. The lane is continue-on-error: true and the existing required checks are unchanged. Dispatch selector is strix-wsl, matching the sibling hardware lanes. The probe change is confined to the E2E harness and is a no-op off WSL.

Test plan

  • cargo test -p e2e-cucumber --lib
  • cargo clippy --locked -p e2e-cucumber --all-targets -- -D warnings, cargo fmt --all --check, license/YAML/signature/DCO hooks
  • The lane itself is green on this branch: platform=strix-halo-wsl os=linux gpu=false, 37 scenarios, 3 xfail (failed as expected), 2 XPASS (2 flaky, 0 stale), 0 unexpected failure(s), with the new System inspection recognizes a WSL host scenario passing against the real host.

A defect the lane found on its first run

examine-both-forms-agree-on-gpu (added by #217) fails here: examine --json
reports has_amd_gpu: false with an empty gpus list on a host whose own
summary, in the same document, carries detected_gfx_target: gfx1151. The GPU
probes are routed out on WSL2 because they read KFD and DRM, neither of which
exists there, while the summary reads the target from the Windows-side driver.

EAI-7998 had this half of the defect only second-hand, from the comment in
capability.rs describing it on MI300X; this is the first direct reproduction.
It survives #220 — the failing run's head contains that commit, which unified
the platform answer but not the GPU one.

It is recorded in expectations.toml as an xfail scoped to is_wsl = true, so
the lane reports it rather than failing on it. An xfail and not a skip: the two
forms are supposed to agree here, so this is a defect, not a premise the
platform cannot meet.

Two notes for whoever watches this lane next:

  • dash-managed-service-metrics failed on two of four runs here while passing on every native lane, and passed on re-run. It is not a product defect: the TUI wait was a single hardcoded 30s, and this workflow now puts a third lane on the one physical Strix box, so a frame that renders comfortably on an idle runner can miss the budget while a sibling loads a model. E2E_TUI_TIMEOUT_SECS now overrides it — mirroring E2E_SERVE_TIMEOUT_SECS next door, which exists for exactly this reason — set to 90s on the three lanes that share the machine. A genuine hang still fails, just later, and a 0 or unparseable value falls back to the default rather than removing the bound. Deliberately not a flaky = true xfail row: contention on a shared runner is not a defect in the product, and an xfail entry would tell every later reader that it is.
  • The full local workspace test hook hits pre-existing failures unrelated to this diff (rocm-core's proc_lifecycle::tree_stop_waits_for_descendants and tree_forced_kill_reaches_sigterm_ignoring_descendant, and rocm's therock::extracting_the_sdk_archive_removes_it); all reproduce identically on a clean main checkout in this environment.

@rominf
rominf changed the base branch from main to ci-shared-windows-strix-runners July 23, 2026 14:25
@rominf
rominf marked this pull request as draft July 23, 2026 14:25
@rominf
rominf force-pushed the ci-shared-windows-strix-runners branch from bb89ce9 to cf10418 Compare July 24, 2026 12:23
Base automatically changed from ci-shared-windows-strix-runners to main July 27, 2026 15:14
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from b1083d5 to 3462382 Compare July 28, 2026 08:32
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from 3462382 to 8bff5d1 Compare July 29, 2026 13:51
@rominf rominf changed the title test(e2e): exercise hosted WSL2 test(e2e): run the E2E suite on the self-hosted WSL2 runner Jul 29, 2026
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch 2 times, most recently from b2e03a4 to fbbe21a Compare August 12, 2026 10:19
@rominf
rominf marked this pull request as ready for review August 12, 2026 12:37
@rominf
rominf requested a review from a team as a code owner August 12, 2026 12:37
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from 6645f21 to 1743d74 Compare August 12, 2026 12:51
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from 1743d74 to 3f38f12 Compare August 12, 2026 13:24
@juhovainio

Copy link
Copy Markdown
Collaborator

Reviewed this against the live branch (head 3f38f12), not just the diff. The change itself is test/CI infrastructure only — new WSL2 self-hosted lane, @requires-wsl tag, capability probe keyed on driver_status instead of a bare gfx_target.is_some(), plus docs. I didn't find anything to flag in the diff: the host_has_usable_gpu split is well-motivated and covered by tests for all four native/WSL x ready/not-ready combinations, derive_platform_slug's WSL branch has a dedicated test, and the @requires-wsl/@requires-bare-metal pairing is tested in both directions so they can't silently collapse into one predicate.

One thing worth knowing before merging, not a defect in this PR: the new lane's own CI run currently fails on examine-both-forms-agree-on-gpurocm examine --json's has_amd_gpu disagrees with the human report's detected_gfx_target on this WSL2 box. I traced it: Examination::probe() in examine.rs returns early on WSL before the code that populates e.gpus/has_amd_gpu ever runs, while the human-readable summary's detected_gfx_target comes from a separate WSL-specific path (detect_examine_gfx_target_fast over the Windows driver inventory) that isn't gated the same way. So JSON says no GPU, human says gfx1151, on the same box. This is a real, pre-existing product bug in rocm-core, not something this PR's diff introduces or needs to fix — it's simply the first time this scenario has run on real WSL2 hardware, since this PR is what creates that lane. The job is continue-on-error: true so it doesn't block, but the PR body's stated test-plan result (0 unexpected failures) is now stale relative to current main and worth a mention/follow-up issue so it doesn't get lost.

Looks safe to approve from my side, pending that follow-up (and the still-pending Windows check).

juhovainio
juhovainio previously approved these changes Aug 12, 2026
@rominf rominf closed this Aug 13, 2026
@rominf rominf reopened this Aug 13, 2026
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from cb398d5 to 4dcdaa7 Compare August 13, 2026 06:07
@rominf

rominf commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

@juhovainio — heads-up that this has moved since your approval at 3f38f12d, and one of the changes is CI config beyond this lane. Worth a re-look rather than merging on the earlier review.

Three commits since:

  1. examine-both-forms-agree-on-gpu recorded as a known bug. The lane reproduces EAI-7998 first-hand: examine --json reports has_amd_gpu: false with an empty gpus list on a host whose own summary, in the same document, carries detected_gfx_target: gfx1151. The GPU probes are routed out on WSL2 because they read KFD and DRM, neither of which exists there, while the summary reads the target from the Windows-side driver. Verified to survive fix(wsl): give examine, serve and the probe one answer about WSL2 #220 — the failing run's head contained that commit. It is an xfail scoped to is_wsl = true, not a skip, because the two forms are supposed to agree here.

  2. The TUI wait budget is now host-tunable — and this touches all three Strix lanes, not just WSL. dash-managed-service-metrics failed 2 of 4 runs here while passing on every native lane. The wait was a single hardcoded 30s, and this PR puts a third lane on the one physical Strix box, so a frame that renders comfortably on an idle runner can miss the budget while a sibling loads a model. E2E_TUI_TIMEOUT_SECS overrides it, mirroring E2E_SERVE_TIMEOUT_SECS next door, set to 90s on the three lanes that share the machine.

    This is the part worth your judgement: I set it on all three because the contention is a property of the shared box, but I have only observed the flake on the WSL lane. Narrowing it to WSL only is entirely reasonable if you would rather not touch the native lanes in this PR.

  3. Rebased onto current main (it had gone DIRTY). Non-trivial resolutions: the @requires-no-wsl tag this branch originally added is dropped in favour of the @requires-bare-metal that landed in test(e2e): pin the doctor commands' contracts, failing where they are broken #216 with identical semantics; the docs keep main's newer three-workflow report text from ci(nightly): consolidate the nightly E2E lanes into one report #221; and the stale examine-can-skip-framework-probing xfail row is not resurrected, since feat(examine): let the user choose which framework to probe #218 landed the --framework flag.

Separately: no CI ran on the two previous heads — GitHub never created an Actions check suite for them, and close/reopen did not help. The rebase regenerated the event and checks are running now, so that appears to have been a lost webhook rather than anything in the diff.

@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from 4dcdaa7 to d9ab09d Compare August 13, 2026 06:44
rominf added 6 commits August 14, 2026 11:20
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Target the Ubuntu-under-WSL2 runner on the Strix Halo box (the `wsl`
label; the native Strix Linux runner carries `native`) instead of
registering a throwaway distro on a hosted windows-latest VM.

The lane now mirrors the other self-hosted Linux lane step for step:
stray-serve reclaim, bounded GPU preflight, toolchain bootstrap, shared
runtime pre-warm, and the full suite with no hand filtering. GPU
preflight is advisory here because GPU access under WSL is what this
lane is proving out: when it is unavailable the capability probe
resolves those scenarios to skip and the rest of the suite still runs.

Dispatch selector is `strix-wsl`, matching the sibling hardware lanes.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`has_amd_gpu` was `gfx_target.is_some()`. On WSL the target is read from
the Windows-side driver and is reported whether or not ROCm can reach the
device, so a distro with /dev/dxg but no librocdxg.so advertises gfx1151
while `rocm serve` refuses with "no usable AMD GPU".

Every `@requires-gpu` scenario therefore ran on a host that cannot serve
one and failed on its premise instead of resolving to not-applicable --
which is the whole job of the capability probe. On the self-hosted WSL
runner that surfaced as `serve-absent-gpu-index-rejected`: it asserts the
index-specific rejection, which the GPU-required pre-flight refuses long
before the index is validated. Its own comment already said so, and it
already carried `@requires-gpu`; the gate just never fired.

Take the CLI's own verdict instead: on WSL require
`driver_status: wsl_rocdxg_ready`, which it sets only once /dev/dxg,
dxcore, librocdxg.so and its ldconfig entry are all in place. Native
hosts are unaffected.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL lane reproduces EAI-7998 first-hand: `examine --json` reports
`has_amd_gpu: false` with an empty `gpus` list on a host whose own
summary, in the same document, carries `detected_gfx_target: gfx1151`.
The GPU probes are routed out on WSL2 because they read KFD and DRM,
neither of which exists there, while the summary reads the target from
the Windows-side driver.

The ticket had this half of the defect only second-hand, from the comment
in capability.rs describing it on MI300X. This lane is the first to hit
it directly, which is the lane doing its job.

Recorded as an xfail rather than a skip: the two forms are supposed to
agree here, so it is a defect, not a premise the platform cannot meet.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…y does

`dash-managed-service-metrics` failed on two of four WSL lane runs while
passing on every native lane, and passed on re-run. The assertion is not
wrong: the TUI wait is a single hardcoded 30s, and this workflow now puts
a third lane on the one physical Strix box, so a frame that renders well
inside 30s on an idle runner can miss it while a sibling loads a model.

`E2E_TUI_TIMEOUT_SECS` overrides it, mirroring `E2E_SERVE_TIMEOUT_SECS`
next door -- same reasoning, that the budget is what varies by host, not
the assertion. Set to 90s on the three lanes that share the machine. A
genuine hang still fails, just later; 0 or an unparseable value falls back
to the default rather than disabling the bound.

Recorded here rather than as a `flaky = true` xfail row: contention on a
shared runner is not a defect in the product, and an xfail entry would
tell every later reader that it is.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL2 lane uploads `e2e-gpu-strix-wsl-report`, but nothing taught the
report tooling that name. It would have passed through
`fallback_descriptor` and rendered as "Gpu Strix Wsl" on Linux — a WSL2
host reported as native Linux, the exact miscoloring the artifact-name
contract exists to prevent.

Teach both halves: `parse_descriptor` maps the artifact to Strix Halo /
WSL2, and `label_for_root_report` maps the `strix-halo-wsl` slug back to
it so a flattened single-artifact download labels the same way. A bare
`wsl` slug (WSL host, no GPU) deliberately stays neutral rather than
claiming hardware the run never saw.

Add the matching nightly lane too: the nightly grid is only comparable
to the per-PR grid while both publish the same platforms, which is what
the contract test asserts.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the worktree-test-hosted-wsl-e2e branch from d9ab09d to 0894e7d Compare August 14, 2026 12:05
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Note for reviewers: this repo doesn't dismiss approvals on push, so the approval at 3f38f12 is still attached but the branch has moved since. What changed:

Rebased onto current main (4eebf1d), no conflicts.

One new commit (ci(e2e): give the WSL2 lane a real column in the report grid). The WSL2 lane uploads e2e-gpu-strix-wsl-report, but nothing taught the report tooling that name, so the two artifact-contract tests added in #221 failed and reddened both Test (affected crates) and windows-build-and-test. Four coordinated edits:

  • CANONICAL_REPORT_ARTIFACTS now lists the artifact.
  • parse_descriptor maps it to Strix Halo / WSL2. Without this the name passes the list check but falls through fallback_descriptor and renders as "Gpu Strix Wsl" on Linux — a WSL2 host reported as native Linux.
  • label_for_root_report maps the strix-halo-wsl slug back to that artifact name, so a flattened single-artifact download labels identically. A bare wsl slug (WSL host, no GPU) deliberately stays neutral instead of claiming hardware the run never saw; both paths are covered by tests.
  • nightly.yml gains the matching WSL lane, since the second contract test asserts that nightly and the per-PR self-hosted workflow publish the same platform set.

Verified locally: cargo fmt --all --check, both CI clippy invocations, the two contract tests, and cargo test --workspace --all-targets. The only failure there is the pre-existing therock::tests::extracting_the_sdk_archive_removes_it parallel-load flake — passes in isolation, outside this diff, and CI's process-per-test nextest doesn't hit it.

The earlier red E2E tests check was an unrelated infra flake reproduced on another branch; the rebase should clear it.

@rominf
rominf dismissed juhovainio’s stale review August 14, 2026 12:30

Dismissing automatically: the head has moved with substantive changes since this approval, and this repo does not dismiss stale reviews on push. See the change-since-review comment on this PR for what differs. Re-review requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants