XDC bug: Adding new cluster requiring setting the active cluster to update the domain cache

[longquanzheng]

Version of Cadence server, and client(which language)
This is very important to root cause bugs.

• Server version: 0.21.3
• Client version: N/A
• Client langauge:N/A

Describe the bug
Domain cache is not updated after adding a new cluster.
To mitigate, I have to set the active cluster name to its current value.

To Reproduce
Is the issue reproducible?

• Yes

Steps to reproduce the behavior:

Expected behavior
The 2nd step shouldn't be needed

Screenshots

(master)$cadence --ct 100 --address <>:7933  --do workflow-prototype d desc
Name: workflow-prototype
UUID: 52d8d0f6-f5aa-4e1d-8d4d-7a9e91255c9a
Description:
OwnerEmail:
DomainData: map[]
Status: REGISTERED
RetentionInDays: 1
EmitMetrics: true
ActiveClusterName: active
Clusters: active, awscmhqa2
HistoryArchivalStatus: DISABLED
VisibilityArchivalStatus: DISABLED
Bad binaries to reset:
+-----------------+----------+------------+--------+
| BINARY CHECKSUM | OPERATOR | START TIME | REASON |
+-----------------+----------+------------+--------+
+-----------------+----------+------------+--------+
qlong@~/indeed/cadence-sandbox-helm-chart:
(master)$cadence --ct 100 --address <>:7933  --do workflow-prototype workflow cancel -w template:indeed:agg:job-id:1653:15
Error: Cancel workflow failed.
Error Details: DomainNotActiveError{Message: Domain: 52d8d0f6-f5aa-4e1d-8d4d-7a9e91255c9a is active in cluster: awscmhqa2, while current cluster active is a standby cluster., DomainName: 52d8d0f6-f5aa-4e1d-8d4d-7a9e91255c9a, CurrentCluster: active, ActiveCluster: awscmhqa2}

Additional context
Add any other context about the problem here, E.g. Stackstace, workflow history.