Skip to content

test(e2e): show stderr when a step fails on a CLI exit code - #248

Open
rominf wants to merge 1 commit into
mainfrom
test/e2e-surface-stderr-on-cli-failure
Open

test(e2e): show stderr when a step fails on a CLI exit code#248
rominf wants to merge 1 commit into
mainfrom
test/e2e-surface-stderr-on-cli-failure

Conversation

@rominf

@rominf rominf commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Summary

E2E steps that assert on rocm's exit code printed only stdout — and rocm
reports its failures on stderr. So a failed serve panicked with:

Step panicked. Captured output: rocm serve failed:

…and nothing after the colon. That is the real state of the MI300X lane today:
a red scenario that cannot be diagnosed from CI logs at all (#247).

run_rocm has returned (stdout, stderr, rc) all along — the five assertion
sites simply bound stderr to _:

let (stdout, _, rc) = crate::run_rocm(world, &[...]);
assert!(rc == 0, "rocm serve failed:\n{stdout}");

They now go through a run_rocm_ok helper that panics with a shared
cli_failure_report: the invocation, the exit code, and both streams, each
labelled and each marked (empty) rather than omitted. That last detail is the
point — "the CLI said nothing" and "the harness dropped the output" are
different diagnoses, and the current message can't tell them apart.

The same failure now reads:

`rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0 --engine lemonade --managed` failed (rc=1)
--- stdout: (empty) ---
--- stderr ---
<the reason>

Two serve steps defer their rc assertion to a later Then step
(assert_vllm_default), so they now carry stderr in the world next to the rc
they already store — cli_stderr is an existing E2eWorld field that several
other steps already populate, so this follows the established pattern rather
than inventing one.

Why the formatter lives in the library

tests/e2e-cucumber/src/lib.rs, not the harness target. The e2e target sets
test = false, so unit tests written there would never run; the library is
covered by the existing cargo test -p e2e-cucumber --lib job. Putting the pure
formatting logic there is what makes it testable at all.

Scope

Only the sites that panic on a CLI failure. Steps that deliberately record
rc for a later content assertion (diagnose, examine) are untouched — they
aren't failing on the exit code, and their Then steps have their own
stderr-capturing When counterparts.

This does not fix the MI300X or Strix Halo Windows failures in #247. It is the
prerequisite: right now there is no way to see why the MI300X serve exits
non-zero. I don't have the hardware to reproduce either failure, so this is
deliberately the part that can be done and verified without it.

Risk: low — test-harness only, no product code, no behavior change to any
passing scenario.

Test plan

Three unit tests on cli_failure_report, the first pinning the exact regression
(empty stdout, populated stderr → the reason survives into the panic):

test tests::failure_report_shows_stderr_when_stdout_is_empty ... ok
test tests::failure_report_marks_empty_streams_explicitly ... ok
test tests::failure_report_keeps_both_streams_when_both_are_present ... ok
test result: ok. 73 passed; 0 failed

Also run locally: cargo clippy --locked --workspace --all-targets -- -D warnings,
cargo clippy --locked -p e2e-cucumber --test e2e -- -D warnings (the harness is
excluded from the first, so both are needed), and
prek run --all-files --no-group local-tools.

Being straight about one thing: these tests cover a new function, so there is no
"fails before, passes after" — the old code path had nothing to assert on. The
before/after evidence is the message shape shown above. The end-to-end proof is
the next MI300X failure actually printing a reason, which needs a self-hosted run.

Closes nothing on its own; unblocks #247.

  • Not a bug fix in product code; no tests/e2e-cucumber/expectations.toml xfail rows to narrow.

@rominf

rominf commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator Author

CI status, with evidence

E2E tests (mock lane) is red, and it is not this change. Three attempts of the
same job on this commit failed two different scenarios:

attempt unexpected failure
1 eai-7960-gen-tps-expiry-boundary
2 eai-7960-gen-tps-expiry-boundary
3 engine-shell-marks-the-prompt

A varying failure set is a flaky lane, not a deterministic break — and
engine-shell-marks-the-prompt is the same scenario that flaked on main in run
31629990566. Both failing scenarios are PTY/TUI screen-scrape assertions with
fixed timeouts.

Running the full mock suite locally on this exact commit (6708e74) is clean:

42 scenarios (39 passed, 3 failed)
Reconciliation: 3 xfail (failed as expected), 0 XPASS, 0 unexpected failure(s)

The 3 failures are the expected xfails. eai-7960-gen-tps-expiry-boundary passes
locally both in the full suite and on its own.

I'll also note the diff cannot reach either scenario: both are driven entirely by
dash_steps.rs / chat steps and the mock server, and this change only touches
serve/install step helpers. Happy to be shown wrong on that.

E2E tests (GPU) and E2E tests (Strix Halo, Windows) are red identically to
main
— 46/7 with 6 xfail on GPU, 30/5 with 4 xfail on Windows, one unexpected
failure in each, both serve-hf-checkpoint-inference (#247).

The GPU lane demonstrates what this PR is for

That previously-undiagnosable failure now reads:

`rocm serve unsloth/Qwen3-0.6B-GGUF:Q4_0 --engine lemonade --managed` failed (rc=1)
--- stdout: (empty) ---
--- stderr ---
...
Error: request_failed: Lemonade server did not become ready: exit status: 1;
.../lemond: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found

Root cause in one line, on the first run: the Lemonade embeddable 11.5.1 binary
needs glibc 2.38 and the MI300X runner's is older. Before this change the same
failure printed rocm serve failed: and nothing else. Details in #247.

Note --- stdout: (empty) --- doing its job — the CLI really did print nothing to
stdout, which is what made the old message look like lost output.

Steps that assert on rocm's exit code printed only stdout, and rocm
reports its failures on stderr. A failed serve therefore panicked with
"rocm serve failed:" followed by nothing at all, which is how the
MI300X lane ended up with a red scenario nobody can diagnose from CI
(EAI-8031).

run_rocm returns stderr already -- the five assertion sites just bound
it to `_`. Replace them with a run_rocm_ok helper that panics through a
shared cli_failure_report: invocation, exit code, and both streams, each
labelled, each marked "(empty)" rather than omitted. The distinction
matters -- "the CLI said nothing" and "the harness dropped the output"
are different diagnoses, and telling them apart is the whole point.

Two serve steps defer their rc assertion to a later Then step, so they
now carry stderr in the world alongside the rc they already store.

The formatter lives in the library, where `cargo test -p e2e-cucumber
--lib` covers it in CI; the harness target's own tests never run.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant