OCPBUGS-59757: OCPBUGS-59758: Improve MCN test stability

[isabella-janssen]

This is a manual cherry-pick of #29981, since the automated cherry-pick never completed.

---

Work included:
OCPBUGS-59203
This updates the MCN happy scope path test, Should properly update the MCN from the associated MCD, to be a part of the serial test suite instead of the parallel test suite. This update is necessary since the test makes changes to MCN objects that can cause interference with other MCN tests when run in parallel.

OCPBUGS-57524
This does two things to improve the stability of the slow, disruptive MCN tests

• Increase the node readiness timeout to reduce failures in the Should properly create and remove MCN on node creation and deletion due to the newly provisioned node not becoming ready within two minutes.
• Add debugging logs to help with troubleshooting in the occasionally flaky Should properly report MCN conditions on node degrade test.

Test locally:
To run the updated tests locally, run them against any 4.19 cluster with the following command.

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Suite:openshift/conformance/serial][Serial]Should properly update the MCN from the associated MCD [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly report MCN conditions on node degrade [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly create and remove MCN on node creation and deletion [apigroup:machineconfiguration.openshift.io]"

Test with payload:
All MCN tests should consistently pass in the MCO disruptive test suite & default serial test suite.

• Disruptive suite: https://pr-payload-tests.ci.openshift.org/runs/ci/8f303eb0-6888-11f0-93ac-4d919f688b32-0
• Default serial: https://pr-payload-tests.ci.openshift.org/runs/ci/92b81070-6889-11f0-83f3-d2115506bc53-0

[isabella-janssen]

/payload-job periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-aws-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-azure-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-gcp-mco-disruptive-techpreview periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-vsphere-mco-disruptive-techpreview

[openshift-ci]

@isabella-janssen: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-aws-mco-disruptive-techpreview
• periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-azure-mco-disruptive-techpreview
• periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-gcp-mco-disruptive-techpreview
• periodic-ci-openshift-machine-config-operator-release-4.19-periodics-e2e-vsphere-mco-disruptive-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8f303eb0-6888-11f0-93ac-4d919f688b32-0

[isabella-janssen]

/cherry-pick jira OCPBUGS-59203

[openshift-cherrypick-robot]

@isabella-janssen: once the present PR merges, I will cherry-pick it on top of jira in a new PR and assign it to you.

In response to this:

/cherry-pick jira OCPBUGS-59203

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[isabella-janssen]

/jira cherry-pick OCPBUGS-59203

[openshift-ci-robot]

@isabella-janssen: Jira Issue OCPBUGS-59203 has been cloned as Jira Issue OCPBUGS-59757. Will retitle bug to link to clone.
/retitle OCPBUGS-59757: Improve MCN test stability

In response to this:

/jira cherry-pick OCPBUGS-59203

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-59757, which is invalid:

• expected dependent Jira Issue OCPBUGS-59203 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #29981, since the automated cherry-pick never completed.

---

Work included:
OCPBUGS-59203
This updates the MCN happy scope path test, Should properly update the MCN from the associated MCD, to be a part of the serial test suite instead of the parallel test suite. This update is necessary since the test makes changes to MCN objects that can cause interference with other MCN tests when run in parallel.

OCPBUGS-57524
This does two things to improve the stability of the slow, disruptive MCN tests

• Increase the node readiness timeout to reduce failures in the Should properly create and remove MCN on node creation and deletion due to the newly provisioned node not becoming ready within two minutes.
• Add debugging logs to help with troubleshooting in the occasionally flaky Should properly report MCN conditions on node degrade test.

Test locally:
To run the updated tests locally, run them against any 4.19 cluster with the following command.

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Suite:openshift/conformance/serial][Serial]Should properly update the MCN from the associated MCD [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly report MCN conditions on node degrade [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly create and remove MCN on node creation and deletion [apigroup:machineconfiguration.openshift.io]"

Test with payload:
All MCN tests should consistently pass in the MCO disruptive test suite & default serial test suite.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/jira cherry-pick OCPBUGS-57524

[openshift-ci-robot]

@isabella-janssen: Jira Issue OCPBUGS-57524 has been cloned as Jira Issue OCPBUGS-59758. Will retitle bug to link to clone.
/retitle OCPBUGS-59758: OCPBUGS-59757: Improve MCN test stability

In response to this:

/jira cherry-pick OCPBUGS-57524

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-59758, which is invalid:

• expected dependent Jira Issue OCPBUGS-57524 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #29981, since the automated cherry-pick never completed.

---

Work included:
OCPBUGS-59203
This updates the MCN happy scope path test, Should properly update the MCN from the associated MCD, to be a part of the serial test suite instead of the parallel test suite. This update is necessary since the test makes changes to MCN objects that can cause interference with other MCN tests when run in parallel.

OCPBUGS-57524
This does two things to improve the stability of the slow, disruptive MCN tests

• Increase the node readiness timeout to reduce failures in the Should properly create and remove MCN on node creation and deletion due to the newly provisioned node not becoming ready within two minutes.
• Add debugging logs to help with troubleshooting in the occasionally flaky Should properly report MCN conditions on node degrade test.

Test locally:
To run the updated tests locally, run them against any 4.19 cluster with the following command.

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Suite:openshift/conformance/serial][Serial]Should properly update the MCN from the associated MCD [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly report MCN conditions on node degrade [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly create and remove MCN on node creation and deletion [apigroup:machineconfiguration.openshift.io]"

Test with payload:
All MCN tests should consistently pass in the MCO disruptive test suite & default serial test suite.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/payload-job periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-single-node-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

[openshift-ci]

@isabella-janssen: trigger 5 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

• periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial
• periodic-ci-openshift-release-master-nightly-4.19-e2e-aws-ovn-single-node-techpreview-serial
• periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial
• periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial
• periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/92b81070-6889-11f0-83f3-d2115506bc53-0

[openshift-ci]

@isabella-janssen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-etcd-scaling	eea68de	link	false	/test e2e-aws-ovn-etcd-scaling
ci/prow/e2e-metal-ipi-ovn-kube-apiserver-rollout	eea68de	link	false	/test e2e-metal-ipi-ovn-kube-apiserver-rollout
ci/prow/e2e-openstack-serial	eea68de	link	false	/test e2e-openstack-serial
ci/prow/e2e-azure-ovn-upgrade	eea68de	link	false	/test e2e-azure-ovn-upgrade
ci/prow/e2e-vsphere-ovn-etcd-scaling	eea68de	link	false	/test e2e-vsphere-ovn-etcd-scaling
ci/prow/e2e-gcp-disruptive	eea68de	link	false	/test e2e-gcp-disruptive
ci/prow/e2e-openstack-ovn	eea68de	link	false	/test e2e-openstack-ovn
ci/prow/e2e-metal-ipi-serial	eea68de	link	false	/test e2e-metal-ipi-serial
ci/prow/e2e-metal-ipi-virtualmedia	eea68de	link	false	/test e2e-metal-ipi-virtualmedia
ci/prow/e2e-vsphere-ovn-dualstack-primaryv6	eea68de	link	false	/test e2e-vsphere-ovn-dualstack-primaryv6
ci/prow/e2e-azure-ovn-etcd-scaling	eea68de	link	false	/test e2e-azure-ovn-etcd-scaling
ci/prow/e2e-aws-ovn-edge-zones	eea68de	link	true	/test e2e-aws-ovn-edge-zones
ci/prow/e2e-gcp-fips-serial	eea68de	link	false	/test e2e-gcp-fips-serial
ci/prow/e2e-gcp-ovn	eea68de	link	true	/test e2e-gcp-ovn
ci/prow/e2e-aws-disruptive	eea68de	link	false	/test e2e-aws-disruptive
ci/prow/e2e-metal-ipi-ovn-ipv6	eea68de	link	true	/test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-gcp-ovn-etcd-scaling	eea68de	link	false	/test e2e-gcp-ovn-etcd-scaling

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[yuqi-zhang]

The diff looked a bit weird, but cross-comparing the two branches the content seem to align, so

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: isabella-janssen, yuqi-zhang
Once this PR has been reviewed and has the lgtm label, please assign dgoodwin for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[isabella-janssen]

/jira refresh

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-59758, which is invalid:

• expected dependent Jira Issue OCPBUGS-57524 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/jira refresh

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-59757, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.z) matches configured target version for branch (4.19.z)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-59203 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-59203 targets the "4.20.0" version, which is one of the valid target versions: 4.20.0
• bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

This is a manual cherry-pick of #29981, since the automated cherry-pick never completed.

---

Work included:
OCPBUGS-59203
This updates the MCN happy scope path test, Should properly update the MCN from the associated MCD, to be a part of the serial test suite instead of the parallel test suite. This update is necessary since the test makes changes to MCN objects that can cause interference with other MCN tests when run in parallel.

OCPBUGS-57524
This does two things to improve the stability of the slow, disruptive MCN tests

• Increase the node readiness timeout to reduce failures in the Should properly create and remove MCN on node creation and deletion due to the newly provisioned node not becoming ready within two minutes.
• Add debugging logs to help with troubleshooting in the occasionally flaky Should properly report MCN conditions on node degrade test.

Test locally:
To run the updated tests locally, run them against any 4.19 cluster with the following command.

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Suite:openshift/conformance/serial][Serial]Should properly update the MCN from the associated MCD [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly report MCN conditions on node degrade [apigroup:machineconfiguration.openshift.io]"

./openshift-tests run-test "[Suite:openshift/machine-config-operator/disruptive][sig-mco][OCPFeatureGate:MachineConfigNodes] [Serial][Slow]Should properly create and remove MCN on node creation and deletion [apigroup:machineconfiguration.openshift.io]"

Test with payload:
All MCN tests should consistently pass in the MCO disruptive test suite & default serial test suite.

• Disruptive suite: https://pr-payload-tests.ci.openshift.org/runs/ci/8f303eb0-6888-11f0-93ac-4d919f688b32-0
• Default serial: https://pr-payload-tests.ci.openshift.org/runs/ci/92b81070-6889-11f0-83f3-d2115506bc53-0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@isabella-janssen: This pull request references Jira Issue OCPBUGS-59757, which is valid.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.19.z) matches configured target version for branch (4.19.z)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-59203 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
• dependent Jira Issue OCPBUGS-59203 targets the "4.20.0" version, which is one of the valid target versions: 4.20.0
• bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[isabella-janssen]

/hold

Letting this sit for a bit longer since it is not super urgent & having more history on the success of the tests in 4.20 will build more confidence in the fix.