soft delete based on service endpoints

[maleck13]

Blocked by: #356 , #367

What

Currently healthchecks have to be defined and use a HTTP request response model to check the health. This is good for checking the end to end connectivity. However it requires some effort from the developer and the platform engineer in order to get into a good state and it will not work for a wildcard listener. Ideally we would have a default health check that could be performed without the need for additional configuration.

I would like that the DNS operator on a given cluster automatically checked for available endpoints via an endpointslice for a given backend ref if that backend ref was of type Service. If there were no available endpoints across all services associated with a listener, it should label those gateways addresses with soft_delete to attempt removal from the DNS Response.

Additional Context

Each Listener on a gateway that results in a DNSRecord has at least one HTTPRoute attached. We can inspect the attached HTTPRoutes for a given listener and check each of their backend refs for available endpoints. If there are available endpoints for given listener then we publish or keep the record in place. If there are none we remove the the records for that gateway listener host. We can achieve this without need to make http requests to an application and instead just use the k8s API.

If we chose to go with this, it should be on by default but be something that can be disabled explicitly via the policy for a gateway or listener.

Tests
TBD

[maleck13]

@philbrookes interested in your thoughts on this

[maleck13]

Requires #356