shard-manager: only allow alive pods to register

[yoohaemin]

Currently, a pod that is not healthy can still be registered to shard-manager, but will be deregistered immediately following a healthcheck. This is not desirable as it can cause rapid rebalancing of many shards, and in the worst case, service disruption when the api pods do not get the unassignment notification.

This PR aims to prevent this by not allowing assignment of shards when the shard manager does not find the pod healthy.

Background:

As discussed here: https://discord.com/channels/629491597070827530/1017210544295522325/1296046997601652799

On staging env, we had the problem where shard-manager was looking at a reused ip address. I added the label selector on shard-manager and deployed to prod. The app seemingly behaved normally, but the label selector was wrong, so it was not getting any pods.
What it actually did was: because there were no pods, each app got all shard assigned and then (after shard-manager couldn't find the pod on k8s) immediately unassigned all shards, but the app didn't see that unassign for some reason. Each of the pods thought they held all shards, and it caused some data corruption.
I think I can contribute: on registration, shard-manager checks if the pod is alive first, and refuse to assign any shard in case it doesn't see the pod. In this case, the label mistake will have blown up on staging first, so the bug wouldn't have sneaked into prod.

[ghostdogpr]

Looks good!