CLM should abandon outdated cluster updates

[linki]

Current behaviour

Once CLM has detected an outdated cluster and starts to perform a cluster update, it will be stuck on that particular update until it's finished or CLM is restarted.

Problem

This can lead to issues when a problematic cluster update is in progress and needs to be fixed. After the fix is applied to our config repository or submitted via changes in Cluster Registry it's not going to be picked up by CLM unless CLM is restarted manually.

For example, configuring a wrong Ubuntu AMI will never allow CLM to finish the update. When the wrong AMI setting is fixed, CLM will not pick up that new configuration as it's still stuck in applying the old broken version.

Proposal

CLM should abandon on-going updates if there's already a new update available in the meantime. Similar to a build system, where an outdated in-progress build for the master branch is preempted in favour of the latest HEAD of the master branch that was just merged in.

There might be cases where it's desirable to first finish the current update and then start with the next update. However, CLM is mostly designed and used to not make such assumptions. Furthermore, we already assume that CLM can be restarted or fail and any point simulating the desired behaviour. Therefore, it seems safe to assume that such logic can be part of CLM itself.