Implement changes to plan to use soft delete flag to remove records

[maleck13]

Why

Currently we can blindly remove endpoints without any real safety check to ensure that we wont deliver an NXDomain. So we currently avoid situations where that might happen (see below)

What

There are several scenarios in which an endpoint should be removed from the DNS response but we do not do it currently.

• Gateway is not in a programmed state
• There are no endpoints behind the backend service
• The HTTP based health checks are failing

We need a common piece of logic that will allow us to "remove if not the last one".

How

The following labels will be added to the DNS Operator plan considerations:

• soft_delete
• stop_soft_delete

A soft_delete label indicates that this record should not be present in the zone, unless removing it would also result in the removal of a record with a stop_soft_delete label.

The stop_soft_delete label would be applied for example to GEO CNAMEs by the kuadrant-operator.

The root host is assumed to have a stop_soft_delete label, even if it is not present.

During the processing of the plan in the dns operator, we need to propagate the soft_delete labels through the DNS tree using the following logic.

• If a node has the soft_delete label, add this label to all of it's children.
• If a node has the soft_delete label, check it's parent:
  • If all the parents children have soft_delete label, add it to the parent recursively
• If the soft_delete label would be added to a node with the stop_soft_delete label, do not add it, and remove the label from all of this nodes children and parents recursively.
• If the soft_delete label would be added to the root host of the record, remove it from the entire record.

Once propagation by the above rules is completed, if any soft_delete labels remain, they are processed as if they are delete requests.