Poll: how to fix e2e-test-kubevirt-aws hitting the 2h Prow step timeout

[kaovilai]

Context: while validating PR #2404, ci/prow/5.0-e2e-test-kubevirt-aws failed by hitting Prow's default 2h step timeout (sigs.k8s.io/prow/pkg/entrypoint.DefaultTimeout, no override set in this job's openshift/release config).

Update: the spec that hits CNV-85377/CNV-89684 (checkpoint deletion with maxIncrementalBackups=0) is already scaffolded as a ginkgo.PIt (compile-time pending) in PR #2404 -- it contributes zero runtime, since it never executes. Recent runs confirm this: no occurrence of that freeze pattern, yet the job still hits the 2h ceiling. The real breakdown, from an actual recent run's per-spec durations (test-execution phase only, which is what the 2h ceiling applies to):

• 8 non-kdm CSI/CSI+datamover specs (this job runs TEST_VIRT=true, the full virt suite, not scoped to kdm): ~326-738s each, summing to ~60 min
• "full backup, incremental chain, restart, and max-limit fallback" (does 4 sequential backups in one It): ~24.5 min
• "restore run-state flip..." and "restore a multi-PVC VM..." (PR test: unpend two kdm restore PIts, fixing bugs found via live e2e validation #2404's two newly-unpended kdm specs, previously PIt, now real): ~10 min and ~24 min respectively

Total: ~1h55m of test-execution alone, already brushing the ~120min ceiling before counting cluster-install time separately -- and PR #2404's two newly-real specs are the straw that pushes some runs over. This is plain additive overrun from legitimate spec runtime, not any single stuck/flaking spec.

Vote by reacting 👍 on the option(s) you'd support (multiple OK if you see them as complementary):

A — Bump the timeout
Add timeout: 3h0m0s to this job's test step in openshift/release. Minimal, fast, no test-code risk. Doesn't address the underlying runtime growth — just buys headroom.

B — Split into two jobs
e2e-test-kubevirt-aws (non-kdm CSI, ~60m) + a new kdm-only job (TEST_VIRT_KDM=true, ~50m+ and growing as more kdm specs get unpended). Both comfortably clear the 2h default with no override needed, and CI feedback arrives in parallel instead of serially. Costs a second job's cluster-bootstrap overhead (~47m) and more openshift/release config to maintain.

C — Eliminate redundant coverage
Audit whether the 9 non-kdm CSI specs are already covered by another virt CI job; if so, drop them from this job. Best long-term fix (real time saved, not just a bigger budget) but needs the audit done first.

D — Fail fast on the known-unfixable flake
~~CNV-89684 hits every AWS kdm run so far with no retry ever recovering it. Reduce flakeAttempts to 1 (or skip immediately) for the affected specs on AWS instead of paying the ~20m wait each time.~~ Superseded: the CNV-85377/CNV-89684-affected spec is already ginkgo.PIt (pays zero runtime), and recent runs show the timeout is driven by plain additive real-spec runtime, not this flake recurring. No longer a relevant option here.

My updated read: A as an immediate stopgap while C gets investigated, with B as the follow-up if C shows the CSI specs aren't actually redundant elsewhere — but posting for the team to weigh in rather than deciding unilaterally.

Note

Responses generated with Claude

[Joeavaikath]

A until we can happily revisit imho

[kaovilai]

Follow-up planned for once this timeout question is resolved: the checksum-verification race in the kdm restore specs.

Traced the mechanism: ChecksumPVCBlockDeviceRegion (tests/e2e/lib/virt_helpers.go) mounts the restored PVC via a separate helper pod as a raw block device. Since the PVC is RWO block-mode, that pod only succeeds while the VM's own virt-launcher pod hasn't attached it yet. Checked 2/2 recent runs: the restored VM was already Running on the very first status read, immediately after restore completed -- there's no halted window to catch here, no matter how tight the poll. The existing check (virt_backup_restore_suite_test.go:1629, :1849) silently skips the hard data-integrity assertion whenever this happens, which per those two runs is effectively every time -- the core assertion these specs exist to run has likely never actually executed on this cluster.

Planned fix: after confirming restore done, if the VM is Running, explicitly stop it (v.StopVm, already exists), wait for the virt-launcher pod to fully terminate (releasing the PVC), run the checksum with guaranteed exclusive access, then restart it (v.StartVm, already exists) before the existing "verify VM running again" step.

Deferred here rather than landed immediately: it adds real wall-clock time (stop + wait-for-pod-gone + checksum + restart + wait-for-running-again) at both call sites, on a suite already sitting right at this issue's 2h ceiling (~1h58m on the last clean run). Tackling in full once this timeout question lands.

Note

Responses generated with Claude

[kaovilai]

shubham said split is ok. I think split would work well to make its scope similar to that covered by presubmit runs of e2e on kdm repos which do not run all the cases (split to only KDM already)

[kaovilai]

Reopening — this closed via #2423 merging (test-code split only), but the two actual mitigations are both still unmerged:

• oadp-operator: bump e2e-test-kubevirt-aws timeout to 3h release#84229 (3h timeout bump, option A) — was draft, undrafted just now
• oadp: split kubevirt e2e job (5.0) and add Claude failure analysis to kdm-controller/plugin release#84337 (job split, option B) — was draft, undrafted just now (its oadp-dev/5.0 half only depended on #2423, already merged; the oadp-1.6 half still needs #2424)

Confirmed live: PR #2404's ci/prow/5.0-e2e-test-kubevirt-aws failed 2/2 consecutive retests since the checksum-verification fix landed, both killed by Prow's entrypoint grace-period kill at almost exactly the 2h mark (~1h59m runtime each, both times right after "Deleting DPA" mid-AfterSuite cleanup) — not a flake, the suite is genuinely still over budget because neither fix has actually shipped yet.

Note

Responses generated with Claude