Hibernate clusters instead deleting them

[tobiscr]

Description

Instead of deleting a Kyma runtime directly when RuntimeCRs get deleted, we should suspend the Gardener cluster and set the Runtime-CR to disposed. The final deletion request happens delayed via housekeeping logic (either by a separate job or during the time-base reconciliation of all Runtime CRs).

This feature has to cope with several circumstances:

• How to behave with failing Hibernation of a cluster :

  • If the hibernation is blocked by some Webhooks, KIM will delete the webhooks and retry the hibernation.
  • Delete the worker pools will be triggered (this step will reduce costs to the minimum)

  If this is not solving the hanging hibernation, KIM forces the deletion of the cluster after a few hours (timeout: 2 hours).

• A new state in RuntimeCR is required (e.g. disposed)

• For proper billing, we should start supporting new fields in RuntimeCR which indicate when a cluster was provisioned and its deletion requested (see Transition from KEB API to KIM Runtime CR kyma-metrics-collector#89 )

• A housekeeping job required to deleted disposed cluster when retention period is reached

AC:

• KIM is no longer deleting clusters, but instead hibernating them
  • Deletion requests for RuntimeCRs which are not already in disposed status triggering a hibernation of a cluster.
  • The RuntimeCR of a successfully hibernated cluster is getting the status disposed and are marked to be non-billable (see KIM has to mark the RuntimeCR to be billable #547)
  • Deletion of RuntimeCRs which are in disposed mode and have reached a pre-defined retention time (e.g. 3 days) are finally deleted in Gardener
• Retention timeout for disposed clusters is configurable in KIM
• It's possible to block the deletion of a disposed cluster (e.g. by setting an annotation)

Reasons

Support manual cluster recovery and address risk of unintended deleted Kyma runtimes (either by customers, SAP employees or by a software bug).

Attachments

• EPIC: Safe deletion of Kyma Clusters [EPIC] #126
• Hibernated clusters cannot be charged anymore ( Transition from KEB API to KIM Runtime CR kyma-metrics-collector#89)

[tobiscr]

Closing it as we detected several technical disadvantages when using hibernation for preventing unintended cluster deletion. Focusing now on #642 .