Soft delete unhealthy endpoints

[philbrookes]

Blocked by: #356 , #367 #390

Use Case

When a health check is failing, we don't currently remove the unhealthy endpoint from the DNS provider. An unhealthy endpoint should be removed from the DNS provider as long as it wont result in a NXDoman response

When reconciling a DNS Record, label any unhealthy endpoints with "soft_delete"

Test Cases

• New test to ensure DNS response is correct when an unhealthy cluster removed
• New test to ensure DNS response is correct when an unhealthy cluster is identified but not removed (IE removing a whole GEO)

Document

We will want to update the health check documentation to call out this new behaviour including not removing a GEO

[philbrookes]

https://gist.github.com/philbrookes/93a41f36753524a9f2a49ebd077cce3e

[maleck13]

Blocked by : #390