test(e2e): run the E2E suite on the self-hosted WSL2 runner - #141
Conversation
bb89ce9 to
cf10418
Compare
b1083d5 to
3462382
Compare
3462382 to
8bff5d1
Compare
b2e03a4 to
fbbe21a
Compare
6645f21 to
1743d74
Compare
1743d74 to
3f38f12
Compare
|
Reviewed this against the live branch (head One thing worth knowing before merging, not a defect in this PR: the new lane's own CI run currently fails on Looks safe to approve from my side, pending that follow-up (and the still-pending Windows check). |
cb398d5 to
4dcdaa7
Compare
|
@juhovainio — heads-up that this has moved since your approval at Three commits since:
Separately: no CI ran on the two previous heads — GitHub never created an Actions check suite for them, and close/reopen did not help. The rebase regenerated the event and checks are running now, so that appears to have been a lost webhook rather than anything in the diff. |
4dcdaa7 to
d9ab09d
Compare
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Target the Ubuntu-under-WSL2 runner on the Strix Halo box (the `wsl` label; the native Strix Linux runner carries `native`) instead of registering a throwaway distro on a hosted windows-latest VM. The lane now mirrors the other self-hosted Linux lane step for step: stray-serve reclaim, bounded GPU preflight, toolchain bootstrap, shared runtime pre-warm, and the full suite with no hand filtering. GPU preflight is advisory here because GPU access under WSL is what this lane is proving out: when it is unavailable the capability probe resolves those scenarios to skip and the rest of the suite still runs. Dispatch selector is `strix-wsl`, matching the sibling hardware lanes. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
`has_amd_gpu` was `gfx_target.is_some()`. On WSL the target is read from the Windows-side driver and is reported whether or not ROCm can reach the device, so a distro with /dev/dxg but no librocdxg.so advertises gfx1151 while `rocm serve` refuses with "no usable AMD GPU". Every `@requires-gpu` scenario therefore ran on a host that cannot serve one and failed on its premise instead of resolving to not-applicable -- which is the whole job of the capability probe. On the self-hosted WSL runner that surfaced as `serve-absent-gpu-index-rejected`: it asserts the index-specific rejection, which the GPU-required pre-flight refuses long before the index is validated. Its own comment already said so, and it already carried `@requires-gpu`; the gate just never fired. Take the CLI's own verdict instead: on WSL require `driver_status: wsl_rocdxg_ready`, which it sets only once /dev/dxg, dxcore, librocdxg.so and its ldconfig entry are all in place. Native hosts are unaffected. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL lane reproduces EAI-7998 first-hand: `examine --json` reports `has_amd_gpu: false` with an empty `gpus` list on a host whose own summary, in the same document, carries `detected_gfx_target: gfx1151`. The GPU probes are routed out on WSL2 because they read KFD and DRM, neither of which exists there, while the summary reads the target from the Windows-side driver. The ticket had this half of the defect only second-hand, from the comment in capability.rs describing it on MI300X. This lane is the first to hit it directly, which is the lane doing its job. Recorded as an xfail rather than a skip: the two forms are supposed to agree here, so it is a defect, not a premise the platform cannot meet. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
…y does `dash-managed-service-metrics` failed on two of four WSL lane runs while passing on every native lane, and passed on re-run. The assertion is not wrong: the TUI wait is a single hardcoded 30s, and this workflow now puts a third lane on the one physical Strix box, so a frame that renders well inside 30s on an idle runner can miss it while a sibling loads a model. `E2E_TUI_TIMEOUT_SECS` overrides it, mirroring `E2E_SERVE_TIMEOUT_SECS` next door -- same reasoning, that the budget is what varies by host, not the assertion. Set to 90s on the three lanes that share the machine. A genuine hang still fails, just later; 0 or an unparseable value falls back to the default rather than disabling the bound. Recorded here rather than as a `flaky = true` xfail row: contention on a shared runner is not a defect in the product, and an xfail entry would tell every later reader that it is. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
The WSL2 lane uploads `e2e-gpu-strix-wsl-report`, but nothing taught the report tooling that name. It would have passed through `fallback_descriptor` and rendered as "Gpu Strix Wsl" on Linux — a WSL2 host reported as native Linux, the exact miscoloring the artifact-name contract exists to prevent. Teach both halves: `parse_descriptor` maps the artifact to Strix Halo / WSL2, and `label_for_root_report` maps the `strix-halo-wsl` slug back to it so a flattened single-artifact download labels the same way. A bare `wsl` slug (WSL host, no GPU) deliberately stays neutral rather than claiming hardware the run never saw. Add the matching nightly lane too: the nightly grid is only comparable to the per-PR grid while both publish the same platforms, which is what the contract test asserts. Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
d9ab09d to
0894e7d
Compare
|
Note for reviewers: this repo doesn't dismiss approvals on push, so the approval at 3f38f12 is still attached but the branch has moved since. What changed: Rebased onto current One new commit (
Verified locally: The earlier red |
Dismissing automatically: the head has moved with substantive changes since this approval, and this repo does not dismiss stale reviews on push. See the change-since-review comment on this PR for what differs. Re-review requested.
Summary
runs-on: [self-hosted, linux, strix-halo, wsl])@requires-wsl/@requires-no-wslcapability tags, a WSL system-inspection scenario, and a distinct WSL platform identity in the report grid@requires-gpuon the driver verdict rather than the reported gfx target, so a WSL host without ROCm passthrough reports its GPU scenarios as not applicable instead of failing themWhy: WSL-specific behavior had parser and self-test coverage but nothing running inside a real WSL2 host. Nothing is filtered by hand — the capability probe resolves each scenario against the live host.
GPU preflight is advisory on this lane specifically: GPU access under WSL is what the lane is proving out, so a missing
rocm-smiwarns and continues. A GPU that is present but still held by a leftover serve still fails, since that would corrupt the serve scenarios.Capability gating had to be fixed to make the lane honest
has_amd_gpuwasgfx_target.is_some(). Under WSL the gfx target is read from the Windows-side driver and is reported whether or not ROCm can reach the device, so this runner advertisedgfx1151whilerocm serverefused with "no usable AMD GPU".Every
@requires-gpuscenario therefore ran on a host that cannot serve one and failed on its premise — which is precisely what the capability probe exists to prevent. It surfaced asserve-absent-gpu-index-rejected, whose own comment already explained the trap ("on a no-GPU host the GPU-required pre-flight refuses before the index is ever validated") and which already carried@requires-gpu; the gate simply never fired. Eight further scenarios failed the same way and were only absorbed because unrelated platform-independent xfail entries happened to cover them.The probe now takes the CLI's own verdict: on WSL it requires
driver_status: wsl_rocdxg_ready, which the product sets only once/dev/dxg, dxcore,librocdxg.soand its ldconfig entry are all present. Native hosts are unaffected. On this runner the probe went fromgpu=truetogpu=false, the nine premise failures became not-applicable, and the xfail count dropped from eight to three real ones.What this lane covers today
The runner's WSL distro has
/dev/dxgand dxcore but notlibrocdxg.so, sorocm examinereportsdriver_status: wsl_rocdxg_missingand the runtime cannot reach the GPU. The lane therefore currently exercises WSL host detection, the Windows-to-WSL execution boundary, and the whole non-GPU suite, with the GPU scenarios correctly reported as not applicable rather than failed.That is a runner-configuration gap, not a gap in this change: once the passthrough is installed,
driver_statusflips towsl_rocdxg_ready, the probe reportsgpu=true, and the GPU scenarios begin running on this lane with no edit to this PR.docs/ci-hardware-testing.mddocuments what the distro needs for each tier of coverage.Risk: medium. The lane is
continue-on-error: trueand the existing required checks are unchanged. Dispatch selector isstrix-wsl, matching the sibling hardware lanes. The probe change is confined to the E2E harness and is a no-op off WSL.Test plan
cargo test -p e2e-cucumber --libcargo clippy --locked -p e2e-cucumber --all-targets -- -D warnings,cargo fmt --all --check, license/YAML/signature/DCO hooksplatform=strix-halo-wsl os=linux gpu=false, 37 scenarios,3 xfail (failed as expected), 2 XPASS (2 flaky, 0 stale), 0 unexpected failure(s), with the newSystem inspection recognizes a WSL hostscenario passing against the real host.A defect the lane found on its first run
examine-both-forms-agree-on-gpu(added by #217) fails here:examine --jsonreports
has_amd_gpu: falsewith an emptygpuslist on a host whose ownsummary, in the same document, carries
detected_gfx_target: gfx1151. The GPUprobes are routed out on WSL2 because they read KFD and DRM, neither of which
exists there, while the summary reads the target from the Windows-side driver.
EAI-7998 had this half of the defect only second-hand, from the comment in
capability.rsdescribing it on MI300X; this is the first direct reproduction.It survives #220 — the failing run's head contains that commit, which unified
the platform answer but not the GPU one.
It is recorded in
expectations.tomlas an xfail scoped tois_wsl = true, sothe lane reports it rather than failing on it. An xfail and not a skip: the two
forms are supposed to agree here, so this is a defect, not a premise the
platform cannot meet.
Two notes for whoever watches this lane next:
dash-managed-service-metricsfailed on two of four runs here while passing on every native lane, and passed on re-run. It is not a product defect: the TUI wait was a single hardcoded 30s, and this workflow now puts a third lane on the one physical Strix box, so a frame that renders comfortably on an idle runner can miss the budget while a sibling loads a model.E2E_TUI_TIMEOUT_SECSnow overrides it — mirroringE2E_SERVE_TIMEOUT_SECSnext door, which exists for exactly this reason — set to 90s on the three lanes that share the machine. A genuine hang still fails, just later, and a0or unparseable value falls back to the default rather than removing the bound. Deliberately not aflaky = truexfail row: contention on a shared runner is not a defect in the product, and an xfail entry would tell every later reader that it is.rocm-core'sproc_lifecycle::tree_stop_waits_for_descendantsandtree_forced_kill_reaches_sigterm_ignoring_descendant, androcm'stherock::extracting_the_sdk_archive_removes_it); all reproduce identically on a cleanmaincheckout in this environment.