[release-4.15] mcd_config_drift not working properly

[dkhater-redhat]

- What I did
Modified startConfigDriftMonitor() to initialize the mcd_config_drift metric based on the node's current state instead of unconditionally resetting it to 0. When the MCD pod restarts, the function now checks if the node is degraded due to config drift (by examining the state and reason annotations) and sets the metric accordingly.
- How to verify it

1. Trigger config drift on a node by manually modifying a file managed by MachineConfig
2. Verify mcd_config_drift metric is set and node is Degraded
3. Delete the MCD pod running on that node: oc delete pod -n openshift-machine-config-operator
4. After the pod restarts, verify the mcd_config_drift metric is still set (not reset to 0)
5. Fix the config drift by restoring the file to its expected state
6. Verify the metric clears to 0 once drift is resolved
   - Description for the changelog

[openshift-ci-robot]

@dkhater-redhat: This pull request references Jira Issue OCPBUGS-24003, which is invalid:

• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected Jira Issue OCPBUGS-24003 to depend on a bug targeting a version in 4.16.0, 4.16.z and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

- What I did
Modified startConfigDriftMonitor() to initialize the mcd_config_drift metric based on the node's current state instead of unconditionally resetting it to 0. When the MCD pod restarts, the function now checks if the node is degraded due to config drift (by examining the state and reason annotations) and sets the metric accordingly.
- How to verify it

• How to verify it

1. Trigger config drift on a node by manually modifying a file managed by MachineConfig
2. Verify mcd_config_drift metric is set and node is Degraded
3. Delete the MCD pod running on that node: oc delete pod -n openshift-machine-config-operator
4. After the pod restarts, verify the mcd_config_drift metric is still set (not reset to 0)
5. Fix the config drift by restoring the file to its expected state
6. Verify the metric clears to 0 once drift is resolved
   - Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dkhater-redhat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [dkhater-redhat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ptalgulk01]

Pre-merge verified:

Environment setup:
Platform: AWS
Version: 4.15.0-0-2026-02-06-045848-test-ci-ln-p9xq9sb-latest

Verification Steps:

• Edit the kubelet.config in one of worker node to create a config drift

$ oc debug node/ip-10-0-1-108.us-west-1.compute.internal -- chroot /host bash -c "echo '# CONFIG DRIFT TEST PR5574' >> /etc/kubernetes/kubelet.conf"                                                                                                       
  Starting pod/ip-10-0-1-108.us-west-1.compute.internal ...                                                                                                                                                                                         
  To use host binaries, run `chroot /host`                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                              
  Removing debug pod ... 

• Check the MCP is degrade and config drift been produced

 $ oc get mcp
  - lastTransitionTime: "2026-02-10T08:08:40Z"
    message: 'Node ip-10-0-1-108.us-west-1.compute.internal is reporting: "Node ip-10-0-1-108.us-west-1.compute.internal
      upgrade failure. unexpected on-disk state validating against rendered-worker-fd6057550cb8f2e4428261b0eab3f3be:
      content mismatch for file \"/etc/systemd/system/kubelet.service.d/20-logging.conf\"",
      Node ip-10-0-1-108.us-west-1.compute.internal is reporting: "unexpected on-disk
      state validating against rendered-worker-fd6057550cb8f2e4428261b0eab3f3be: content
      mismatch for file \"/etc/systemd/system/kubelet.service.d/20-logging.conf\""'
    reason: ""
    status: "True"
    type: Degraded                                                                                                   
                                                                                                                                                                                                                                                              
$ oc exec machine-config-daemon-4k6np  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 1.7707109151458156e+09

• Restart MCD Pod Restarting the pod should NOT reset the metric to 0.

  $ oc delete pod machine-config-daemon-4k6np                                                                                                                                                                  

Wait for new pod to start
Check if the drift metric persisted across the pod restart:

$ c exec machine-config-daemon-xsr7x  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 1.7707111779752192e+09
                                                                                                                                                                                                            
 $  oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-44e25c1f5dd93d02d9f6fc5e5a3540de   True      False      False      3              3                   3                     0                      137m
worker   rendered-worker-fd6057550cb8f2e4428261b0eab3f3be   False     True       True       3              2                   2                     1                      137m

• Cleanup & Restoration

 $ oc debug node/ip-10-0-1-108.us-west-1.compute.internal -- chroot /host bash -c "sed -i '/# CONFIG DRIFT TEST PR5574/d' /etc/kubernetes/kubelet.conf"                                                                                                     
 Starting pod/ip-10-0-1-108.us-west-1.compute.internal ...                                                                                                                                                                                         
 To use host binaries, run `chroot /host`                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                             
 Removing debug pod ...  

Wait for MCD to reconcile

  $ oc get node ip-10-0-1-108.us-west-1.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo                                                                                                             
  Done                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                              
$  oc exec machine-config-daemon-xsr7x  -- curl -s localhost:8797/metrics 2>/dev/null | grep mcd_config_drift
# HELP mcd_config_drift timestamp for config drift
# TYPE mcd_config_drift gauge
mcd_config_drift 0                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                              
$  oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-44e25c1f5dd93d02d9f6fc5e5a3540de   True      False      False      3              3                   3                     0                      139m
worker   rendered-worker-fd6057550cb8f2e4428261b0eab3f3be   True      False      False      3              3                   3                     0                      139m

/label qe-approved
/verified by @ptalgulk01

[openshift-ci-robot]

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge verified:

Environment setup:
Platform: AWS
Version: 4.15.0-0-2026-02-06-045848-test-ci-ln-p9xq9sb-latest

Verification Steps:

Verified using Claude

• Before editing the kubelet.config

 $ oc get node ip-10-0-111-17.us-east-2.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo                                                                                                             
 Done                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                             
 $ oc exec -n openshift-machine-config-operator machine-config-daemon-zptsj -- curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift                                                                                                                   
 # HELP mcd_config_drift timestamp for config drift                                                                                                                                                                                                          
 # TYPE mcd_config_drift gauge                                                                                                                                                                                                                               
 mcd_config_drift 0 

• Edit the kubelet.config in one of worker node to create a config drift

$ oc debug node/ip-10-0-111-17.us-east-2.compute.internal -- chroot /host bash -c "echo '# CONFIG DRIFT TEST PR5574' >> /etc/kubernetes/kubelet.conf"                                                                                                       
 Starting pod/ip-10-0-111-17us-east-2computeinternal-debug-j5h49 ...                                                                                                                                                                                         
 To use host binaries, run `chroot /host`                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                             
 Removing debug pod ... 

• Check the MCP is degrade and config drift been produced

 $ oc get node ip-10-0-111-17.us-east-2.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo                                                                                                             
 Degraded                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                             
 $ oc get node ip-10-0-111-17.us-east-2.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/reason}' && echo                                                                                                            
 unexpected on-disk state validating against rendered-worker-e1837bca6cfe3c85710be5b9c17ab418: content mismatch for file "/etc/kubernetes/kubelet.conf"                                                                                                      
                                                                                                                                                                                                                                                             
 $ oc exec -n openshift-machine-config-operator machine-config-daemon-zptsj -- curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift                                                                                                                   
 # HELP mcd_config_drift timestamp for config drift                                                                                                                                                                                                          
 # TYPE mcd_config_drift gauge                                                                                                                                                                                                                               
 mcd_config_drift 1.770359684748165e+09  

• Restart MCD Pod Restarting the pod should NOT reset the metric to 0.

 $ oc delete pod -n openshift-machine-config-operator machine-config-daemon-zptsj                                                                                                                                                                            
 pod "machine-config-daemon-zptsj" deleted                                                                                                                                                                          

Wait for new pod to start

 $ oc get pod -n openshift-machine-config-operator -l k8s-app=machine-config-daemon -o wide | grep "ip-10-0-111-17"                                                                                                                                          
 machine-config-daemon-fnr4l   2/2     Running   0          46s   10.0.111.17    ip-10-0-111-17.us-east-2.compute.internal    <none>           <none>                                                                                                                                                                                                                                                                                                                                                                      

Check if the drift metric persisted across the pod restart:

 $ oc exec -n openshift-machine-config-operator machine-config-daemon-fnr4l -- curl -s localhost:8797/metrics 2>&1 | grep mcd_config_drift                                                                                                                   
 # HELP mcd_config_drift timestamp for config drift                                                                                                                                                                                                          
 # TYPE mcd_config_drift gauge                                                                                                                                                                                                                               
 mcd_config_drift 1.770360083354085e+09                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                             
 $ oc get node ip-10-0-111-17.us-east-2.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo                                                                                                             
 Degraded 

• Cleanup & Restoration

 $ oc debug node/ip-10-0-111-17.us-east-2.compute.internal -- chroot /host bash -c "sed -i '/# CONFIG DRIFT TEST PR5574/d' /etc/kubernetes/kubelet.conf"                                                                                                     
 Starting pod/ip-10-0-111-17us-east-2computeinternal-debug-dtrjk ...                                                                                                                                                                                         
 To use host binaries, run `chroot /host`                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                             
 Removing debug pod ...  

Wait for MCD to reconcile

 $ oc get node ip-10-0-111-17.us-east-2.compute.internal -o jsonpath='{.metadata.annotations.machineconfiguration\.openshift\.io/state}' && echo                                                                                                             
 Done                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                             
 $ oc exec -n openshift-machine-config-operator machine-config-daemon-fnr4l -- curl -s localhost:8797/metrics 2>&1 | grep "^mcd_config_drift"                                                                                                                
 mcd_config_drift 0                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                             
 $ oc get mcp                                                                                                                                                                                                                                                
 NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE                                                                             
 master   rendered-master-e02d034a46e6941f939b1a435209c52f   True      False      False      3              3                   3                     0                      94m                                                                             
 worker   rendered-worker-e1837bca6cfe3c85710be5b9c17ab418   True      False      False      3              3                   3                     0                      94m     

/label qe-approved
/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dkhater-redhat]

/verified by @ptalgulk01

had to remove debugging logs

[openshift-ci-robot]

@dkhater-redhat: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

/verified by @ptalgulk01

had to remove debugging logs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[dkhater-redhat]

/retitle [release-4.15] mcd_config_drift not working properly

[openshift-ci-robot]

@dkhater-redhat: No Jira issue is referenced in the title of this pull request.
To reference a jira issue, add 'XYZ-NNN:' to the title of this pull request and request another refresh with /jira refresh.

Details

In response to this:

- What I did
Modified startConfigDriftMonitor() to initialize the mcd_config_drift metric based on the node's current state instead of unconditionally resetting it to 0. When the MCD pod restarts, the function now checks if the node is degraded due to config drift (by examining the state and reason annotations) and sets the metric accordingly.
- How to verify it

1. Trigger config drift on a node by manually modifying a file managed by MachineConfig
2. Verify mcd_config_drift metric is set and node is Degraded
3. Delete the MCD pod running on that node: oc delete pod -n openshift-machine-config-operator
4. After the pod restarts, verify the mcd_config_drift metric is still set (not reset to 0)
5. Fix the config drift by restoring the file to its expected state
6. Verify the metric clears to 0 once drift is resolved
   - Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@dkhater-redhat: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	68b168a	link	false	/test okd-scos-e2e-aws-ovn
ci/prow/okd-scos-e2e-gcp-op	68b168a	link	false	/test okd-scos-e2e-gcp-op
ci/prow/okd-scos-e2e-gcp-ovn-upgrade	68b168a	link	false	/test okd-scos-e2e-gcp-ovn-upgrade

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.