Skip to content

fix: raise the supported Linux minimum to Ubuntu 24.04 - #261

Open
rominf wants to merge 1 commit into
mainfrom
docs/ubuntu-2404-minimum
Open

fix: raise the supported Linux minimum to Ubuntu 24.04#261
rominf wants to merge 1 commit into
mainfrom
docs/ubuntu-2404-minimum

Conversation

@rominf

@rominf rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

The problem

The docs promise a platform one of the two shipped engines cannot run on.

README.md advertised Linux (x86_64) as "Full support, including the live
dashboard and both inference engines"
, and docs/wsl.md listed
"WSL2 with Ubuntu 24.04 or Ubuntu 22.04" as a prerequisite. Together those tell a
22.04 user — native or WSL2 — that both engines work.

They do not. Every published Lemonade embeddable, v10.2.0 through v11.5.2 (all 15
ubuntu-x64 releases), is linked against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu
22.04 ships glibc 2.35 and GLIBCXX_3.4.30, so lemond is refused by the dynamic
loader before it starts. There is no older release to pin back to and no supported
way to put glibc 2.38 on jammy. Full analysis in #258.

This is option 1 from that issue — narrow the documented matrix to match reality.

What changed

One flat minimum: Ubuntu 24.04 or newer, for Linux and WSL2 alike.

  • README.md: both the Linux and WSL2 rows now say "Ubuntu 24.04 or newer", and a
    short paragraph below gives the glibc number behind that line — glibc 2.38 with
    GLIBCXX_3.4.32 — so the requirement still means something to a reader on RHEL,
    SLES, or Fedora who cannot map an Ubuntu version onto their own distribution.
  • docs/wsl.md: prerequisite 2 is now "Ubuntu 24.04 or newer", with the same reason
    stated inline.
  • scripts/wsl_preflight.py: 22.04 dropped from the supported set, and the
    ubuntu check detail now names the minimum. Covered by a new --self-test case
    (an otherwise-ready 22.04 host must fail the check), which fails against the old
    set and passes against the new one, and runs in CI on both the Linux and Windows
    lanes.

The minimum is deliberately stated as a single number rather than split per engine.
The Lemonade floor is the binding one; a per-engine matrix would be more precise but
harder to state and harder to test, and the simpler line is the one being supported.

The one edit a maintainer might want reverted

scripts/wsl_preflight.py is a behaviour change, not documentation — hence the fix:
title rather than docs:. It is included because that script is the automated form of
the docs/wsl.md prerequisite list and is invoked directly from it; leaving it
reporting ubuntu: ok on a host Lemonade cannot run on would relocate the
inconsistency rather than fix it. If you would rather keep this PR strictly to prose,
it is a self-contained three-line revert.

Also found, deliberately not changed

  • apps/rocm/src/main.rs driver-install matrix (("ubuntu", "22.04" | "24.04"),
    codename_for_version). That mirrors AMD's own documented amdgpu-dkms install
    matrix, which is a separate upstream support matrix; rocm install driver still
    produces a correct plan on 22.04 and narrowing it would be a behaviour change beyond
    this issue.
  • crates/rocm-core/src/diagnose.rs suggesting rocm/dev-ubuntu-22.04 as a
    known-good container image. A container brings its own userland, so the host glibc
    claim does not apply; it is not a host-OS support statement.

This does not fix the user experience, only the promise

#258 also asks for a preflight error — "the lemonade engine requires glibc 2.38 or
newer; this host has 2.35"
— instead of a raw loader message. That is deliberately
out of scope here, but worth stating plainly: no such preflight exists today. The
only glibc-aware code in the tree is a low-weight "version `?glibc" keyword in the
rocm diagnose classifier, explicitly annotated "tangential"; there is no glibc check
anywhere in the Lemonade download or launch path. A 22.04 user who has not read these
docs still gets an opaque version `GLIBC_2.38' not found out of a subprocess. The
preflight remains worth doing on its own, and it would be worth tracking separately
rather than letting it lapse when this merges.

Related

Verification

prek run --all-files --no-group local-tools (all passed: whitespace, EOF, yaml,
merge-conflict, line endings, ruff check, ruff format, shellcheck, cargo fmt),
hawkeye check --config licenserc.toml (no missing headers), cargo fmt --all --check (clean), python3 scripts/wsl_preflight.py --self-test (ok; and confirmed
the new assertion still fails against the pre-change supported set).

No Gherkin scenario accompanies this: the rocm/rocmd binaries have no observable
behaviour change. The one behavioural edit is to a standalone helper script, whose
contract is covered by its own --self-test, already run by CI.

Searched tests/e2e-cucumber/expectations.toml for stale xfails tied to this issue —
the Lemonade rows there are all EAI-7423 (managed serve shutting down after ready),
unrelated to the glibc floor, so nothing to remove or narrow.

Fixes #258

@rominf
rominf requested a review from a team as a code owner August 14, 2026 12:15
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

CI status, so the one red check is not left unexplained.

All 16 required checks are green, including prek (lint / hygiene), License header check (hawkeye), Commit signatures + sign-off, E2E tests, and both CodeQL analyses.

E2E tests (GPU) is red, and it is pre-existing on main, not caused by this PR. The lane is advisory (not in the required set). Evidence:

  • The failure is one unexpected failure, serve-hf-checkpoint-inference, failing at "a canonical Hugging Face GGUF checkpoint is being served on lemonade" with rocm serve failed. Reconciliation line: 5 xfail, 0 XPASS, 1 unexpected failure(s).
  • The main-branch run from the same window (job 94765233909) fails with the identical reconciliation line and the identical scenario, so the regression is already on main.
  • This PR changes README.md, docs/wsl.md, and scripts/wsl_preflight.py. None of the three is read by the rocm binary or the E2E harness, and the GPU runner is Ubuntu 24.04.4 LTS — above the new documented minimum either way, so nothing in this change can reach that lane.
  • The symptom is a lemonade serve failure on a headless self-hosted runner, which is what rocm serve --engine lemonade fails on any headless host where XDG_RUNTIME_DIR is unset #259 describes: the runner's glibc 2.39 clears the floor tracked in Lemonade engine cannot start on Ubuntu 22.04 (glibc 2.35), but README and docs/wsl.md advertise it as supported #258, so the XDG_RUNTIME_DIR failure underneath is no longer masked. There is already a fix-lemonade-missing-xdg-runtime-dir branch in flight for it.

E2E tests (Strix Halo, Ubuntu) passed. E2E tests (Strix Halo, Windows) is still queued behind the self-hosted runner backlog; it is also advisory and I will follow up if it reports anything other than the same pre-existing state.

@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Follow-up on the two advisory self-hosted lanes, now that both have finished.

  • E2E tests (Strix Halo, Ubuntu)passed.
  • E2E tests (Strix Halo, Windows)failed, and it is the same pre-existing failure: 3 xfail, 0 XPASS, 1 unexpected failure(s) / serve-hf-checkpoint-inference. The unrelated branch feat/rocm-version-tag-branch-hash (job 94760021263), which touches no docs, produces a byte-identical reconciliation line and the same scenario, so this is repo-wide rather than anything in this PR. main's own Windows lane is still queued behind the runner backlog; its GPU lane already failed the same way (job 94765233909).

So both red lanes fail on serve-hf-checkpoint-inference, on hosts already above the new documented minimum, and both reproduce off this branch. Neither is in the 16 required checks, and all 16 required checks are green.

Every published Lemonade embeddable, v10.2.0 through v11.5.2, is linked
against GLIBC_2.38 and GLIBCXX_3.4.32. Ubuntu 22.04 ships glibc 2.35, so
the Lemonade engine cannot start there at all, yet README.md advertised
Linux x86_64 as "full support ... both inference engines" and docs/wsl.md
named 22.04 as a supported WSL base.

Document a single minimum of Ubuntu 24.04 for Linux and WSL2, keeping the
glibc number as the reason so the requirement stays meaningful on other
distributions, and drop 22.04 from the WSL preflight's supported set so the
automated check matches the prerequisite it enforces.

Fixes #258

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the docs/ubuntu-2404-minimum branch from ceb0d59 to 7b66483 Compare August 14, 2026 13:47
@rominf rominf changed the title docs: raise the documented Linux minimum to Ubuntu 24.04 fix: raise the supported Linux minimum to Ubuntu 24.04 Aug 14, 2026
@rominf

rominf commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator Author

Updated per maintainer decision: one flat minimum of Ubuntu 24.04, no per-engine matrix. The glibc number stays as the reason behind that line so it still means something on a non-Ubuntu distribution. Retitled docs:fix:, since the diff includes the scripts/wsl_preflight.py behaviour change.

Head is now 7b66483, rebased onto current main (0063ef15).

CI on the new head: all 16 required checks green. Re-verified locally after the edit — prek run --all-files --no-group local-tools, hawkeye check, cargo fmt --all --check, and wsl_preflight.py --self-test including the new 22.04 regression case (confirmed it still fails against the pre-change supported set).

E2E tests (GPU) is red again for the same pre-existing serve-hf-checkpoint-inference / #259 reason explained above; the two Strix lanes are still running. All three are advisory, none is in the required set, and none is reachable from a change to two markdown files and a helper script.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Lemonade engine cannot start on Ubuntu 22.04 (glibc 2.35), but README and docs/wsl.md advertise it as supported

1 participant