Skip to content

fix(lemonade): retry interrupted backend setup - #249

Open
volen-silo wants to merge 1 commit into
mainfrom
fix/lemonade-preparation-retry
Open

fix(lemonade): retry interrupted backend setup#249
volen-silo wants to merge 1 commit into
mainfrom
fix/lemonade-preparation-retry

Conversation

@volen-silo

Copy link
Copy Markdown
Collaborator

Summary

  • Retry Lemonade's llama.cpp backend installation once when the backend subprocess fails during first serve.
  • Keep the retry at the backend boundary so deterministic archive, disk, and backend-selection failures are not repeated and a healthy embeddable is not re-extracted.
  • After the retry fails, recommend rocm engines install lemonade --reinstall before retrying rocm serve.
  • Add deterministic unit coverage and a black-box Cucumber scenario for the visible retry count and recovery guidance.

Why: a transient connection loss during first-time Lemonade backend preparation previously aborted rocm serve with a generic error, forcing users to discover the manual recovery sequence themselves.

Risk: low. The change is limited to the backend-install phase, allows one additional attempt, and exposes its failure fixture only in E2E-enabled builds.

Test plan

  • cargo xtask e2e -- -n 'Repeated Lemonade preparation failure'
  • cargo test -p rocm-engine-lemonade backend_install_
  • cargo test -p e2e-cucumber --lib
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace --all-targets -- --skip therock::tests::extracting_the_sdk_archive_removes_it
  • cargo fmt --all -- --check

The skipped archive test is a pre-existing parallel-suite flake on this host (tar temporarily missing from PATH) and passes in isolation.

Signed-off-by: Eugene Volen <Eugene.Volen@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant