NO-JIRA: fix(e2e): lower pull secret in-place propagation test gate to 4.22

[enxebre]

Summary

• Lower the CPOAtLeast gate on the pull secret in-place propagation e2e test from Version423 to Version422, since the CP pull-secret watches were backported to release-4.22 via #8408

Test plan

• The test should now run on 4.22+ clusters instead of being skipped

🤖 Generated with Claude Code

Summary by CodeRabbit

• Tests
  • Updated version compatibility check in end-to-end test validation for pull secret propagation.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@enxebre: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

• Lower the CPOAtLeast gate on the pull secret in-place propagation e2e test from Version423 to Version422, since the CP pull-secret watches were backported to release-4.22 via #8408

Test plan

• The test should now run on 4.22+ clusters instead of being skipped

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: fdbd675c-66ce-429f-bfb8-16f48112c7f7

📥 Commits

Reviewing files that changed from the base of the PR and between 9e283ae and 5e21633.

📒 Files selected for processing (1)

• test/e2e/util/util.go

---

📝 Walkthrough

Walkthrough

The pull request lowers the minimum Control Plane Operator (CPO) version gate for a single end-to-end test. Specifically, the EnsureGlobalPullSecret subtest in test/e2e/util/util.go that verifies in-place updates to hostedCluster.Spec.PullSecret propagate to the guest cluster without triggering a NodePool rollout now calls CPOAtLeast with Version422 instead of Version423. This single-line change allows the test to execute on an earlier CPO version.

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: lowering the test gate version requirement from 4.23 to 4.22 for the pull secret propagation test.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR only changes a version gate (Version423→Version422) in test body logic, not test titles. All EnsureGlobalPullSecret test names are stable strings with no dynamic identifiers.
Test Structure And Quality	✅ Passed	Test code meets all five quality criteria: single responsibility, proper t.Cleanup(), 150s timeouts on Eventually calls, meaningful assertion messages, and consistent patterns.
Microshift Test Compatibility	✅ Passed	PR does not add new Ginkgo e2e tests. It only modifies version gate in existing utility function from Version423 to Version422. Check is not applicable.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	This PR only modifies a version gate (CPOAtLeast Version423→Version422) in an existing test, not adding new Ginkgo e2e tests. The custom check applies only to newly added tests.
Topology-Aware Scheduling Compatibility	✅ Passed	PR modifies only test/e2e/util/util.go, a test file. Custom check applies to deployment manifests, operator code, and controllers. No scheduling constraints introduced.
Ote Binary Stdout Contract	✅ Passed	Change is in a t.Run() subtest, not process-level code. CPOAtLeast() uses only t.Logf()/t.Skipf(), not stdout. No OTE contract violations.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Check applies only to new Ginkgo e2e tests. This PR only lowers an existing test's version gate from Version423 to Version422 with no test logic changes.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.34%. Comparing base (9e283ae) to head (5e21633).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8553   +/-   ##
=======================================
  Coverage   40.34%   40.34%           
=======================================
  Files         755      755           
  Lines       93167    93167           
=======================================
  Hits        37587    37587           
  Misses      52877    52877           
  Partials     2703     2703           

Flag	Coverage Δ
cmd-support	34.30% <ø> (ø)
cpo-hostedcontrolplane	41.76% <ø> (ø)
cpo-other	40.14% <ø> (ø)
hypershift-operator	50.72% <ø> (ø)
other	31.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sdminonne]

/lgtm

[sdminonne]

Not a lot of things to say.

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre, sdminonne

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[cwbotbot]

Test Results

e2e-aws

• Status: ✅ PASS
• Started: 2026-05-21T15:08:29Z
• View Job
• View Job History

e2e-aks

• Status: ✅ PASS
• Started: 2026-05-20T09:33:55Z
• View Job
• View Job History

[enxebre]

/verified by e2e @enxebre

[openshift-ci-robot]

@enxebre: This PR has been marked as verified by e2e @enxebre.

Details

In response to this:

/verified by e2e @enxebre

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[enxebre]

/test e2e-aws

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2057067651628273664 | Cost: $2.977428499999999 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[enxebre]

/test e2e-aws

[hypershift-jira-solve-ci]

I now have all the evidence needed. Here is the complete analysis:

Test Failure Analysis Complete

Job Information

• Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
• Build ID: 2057362866691903488
• Target: e2e-aws
• Failed Step: e2e-aws-hypershift-aws-run-e2e-nested (test phase, 1h8m4s)
• Test Results: 584 tests, 25 skipped, 1 failure (TestCreateClusterCustomConfig)

Test Failure Analysis

Error

hypershift_framework.go:518: failed to create cluster, tearing down: failed to create infra:
cannot create VPC S3 endpoint: operation error EC2: CreateVpcEndpoint, exceeded maximum number
of attempts, 11, https response error StatusCode: 503, RequestID: 1d39c229-24cc-4dd3-b1a2-240073ce7283,
api error RequestLimitExceeded: Request limit exceeded.

Summary

The sole test failure (TestCreateClusterCustomConfig) is caused by AWS EC2 API rate limiting (HTTP 503, RequestLimitExceeded) during VPC S3 endpoint creation. The test suite runs 20 tests in parallel (-test.parallel=20), each creating hosted clusters with full AWS infrastructure (VPCs, endpoints, etc.). With ~28 clusters being provisioned concurrently, the CreateVpcEndpoint API call for the custom-config cluster was throttled by AWS after 11 retry attempts. This failure is completely unrelated to PR #8553, which only changes a version gate constant (Version423 → Version422) in test/e2e/util/util.go for the pull secret propagation test. All other 583 tests passed.

Root Cause

AWS EC2 API rate limiting (transient infrastructure flake).

The TestCreateClusterCustomConfig test failed during cluster infrastructure provisioning — specifically at the CreateVpcEndpoint step when creating an S3 gateway endpoint for the new VPC. The AWS EC2 API returned HTTP 503 with error code RequestLimitExceeded, indicating the account/region had exhausted its API request quota.

Why this happened:

• The e2e test suite runs with -test.parallel=20, meaning up to 20 test functions execute concurrently
• Each test creates a full hosted cluster, which provisions AWS infrastructure including VPCs, subnets, security groups, and VPC endpoints
• In this run, ~28 clusters were successfully created while this single cluster creation failed — the sheer volume of concurrent AWS API calls across all parallel tests caused the CreateVpcEndpoint call for the custom-config cluster to be rate-limited
• The SDK retried 11 times (the maximum configured attempts) but the throttling persisted, causing the test to fail after 108 seconds

Why this is NOT related to PR #8553:

• PR NO-JIRA: fix(e2e): lower pull secret in-place propagation test gate to 4.22 #8553 modifies a single line in test/e2e/util/util.go, changing CPOAtLeast(t, Version423, ...) to CPOAtLeast(t, Version422, ...) for the pull secret in-place propagation test
• The failing test TestCreateClusterCustomConfig is an entirely different test that provisions cluster infrastructure — it has no code path overlap with the pull secret propagation version gate change
• The error occurs in hypershift_framework.go:518 during AWS infrastructure creation, which is common framework code unrelated to the PR

Recommendations

1. Rerun the CI job — This is a transient AWS API rate-limiting failure unrelated to the PR changes. A /retest should pass.
2. No code changes needed — The PR's one-line version gate change (Version423 → Version422) has zero relationship to the CreateVpcEndpoint failure. The 583/584 pass rate confirms the change itself is sound.
3. Known flake pattern — RequestLimitExceeded errors on CreateVpcEndpoint are an inherent risk when running 20 parallel HyperShift e2e tests that each create full AWS infrastructure. This is a well-understood CI environment limitation, not a product bug.

Evidence

Evidence	Detail
Failed Test	TestCreateClusterCustomConfig (108.14s)
Error Code	AWS RequestLimitExceeded on EC2:CreateVpcEndpoint
HTTP Status	503 Service Unavailable
AWS RequestID	1d39c229-24cc-4dd3-b1a2-240073ce7283
Retry Attempts	11 (max exceeded)
Test Parallelism	-test.parallel=20
Clusters Created Successfully	28
Clusters Failed to Create	1 (this test only)
Total Test Results	584 tests, 25 skipped, 1 failure
PR #8553 Change	Single line: Version423 → Version422 in test/e2e/util/util.go
PR Relation to Failure	None — different test, different code path
Failed Step	e2e-aws-hypershift-aws-run-e2e-nested (test phase)
Step Duration	1h8m4s

---

[openshift-merge-bot]

/retest-required

Remaining retests: 0 against base HEAD 36dfb1b and 2 for PR HEAD 5e21633 in total

[openshift-ci]

@enxebre: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.