simulate: retry still-failing scenarios in CI with --retries (default 3) - #953
Open
u9g wants to merge 2 commits into
Open
simulate: retry still-failing scenarios in CI with --retries (default 3)#953u9g wants to merge 2 commits into
u9g wants to merge 2 commits into
Conversation
Pure move: the loop body is unchanged, its cross-run detection state (broken agent, quota warning, peak concurrency) now lives on a struct so a follow-up can poll more than one run per invocation.
…efault 3) After a run finishes with failures, CI mode re-runs only the failing scenarios against the same already-registered agent, up to --retries times; a scenario passes when any attempt does. The scenarios come from the finished run itself, so generated-from-source runs retry without re-uploading or regenerating. Systemic conditions (broken agent, quota exhaustion) are never retried. The verdict (counts, --baseline comparison, exit error) reads each scenario's outcome from the last attempt that ran it to a terminal state, so a cancelled retry cannot launder an earlier failure. Every attempt's transcript still prints; a failure that later passed keeps its transcript but loses its ::error:: annotation, and the final counts name the scenarios that passed on retry so flakes stay visible.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
jason/simulate-ci-baseline(the--baselinefeature, in progress in another session) — only the top two commits are this PR.What
CI runs of
lk agent simulatenow retry failed scenarios instead of failing the build on the first flake. After a run finishes with failures, the CLI re-runs only the still-failing scenarios against the same already-registered agent, up to--retriestimes (default 3,--retries 0disables). A scenario passes when any attempt passes.How
run.scenario_group), which is populated in both--scenariosand generated-from-source modes — so retries never re-upload source or regenerate scenarios, and generated scenarios are retried verbatim rather than nondeterministically regenerated.--baselinecomparison, exit error) reads each scenario's outcome from the last attempt that ran it to a terminal state, so a cancelled retry cannot launder an earlier failure.::error::annotation, and the final counts line names the scenarios that passed on retry.Commits
refactor(simulate): pure move — the CI poll loop intociRunPoller, so one invocation can poll several runs with shared broken-agent/quota detection state.feat(simulate): the--retriesflag and retry loop, with tests for the outcome-merging helpers.