XdsDepManager generated invalid configuration

[ejona86]

The CI for #12141 flaked:

io.grpc.StatusRuntimeException: INTERNAL: CdsLb for cluster0: Unable to find non-dynamic root cluster

However, that is supposed to be impossible:

grpc-java/xds/src/main/java/io/grpc/xds/CdsLoadBalancer2.java

Lines 89 to 95 in d88ef97

	StatusOr<XdsClusterConfig> clusterConfigOr = xdsConfig.getClusters().get(clusterName);
	if (clusterConfigOr == null) {
	if (clusterSubscription == null) {
	// Should be impossible, because XdsDependencyManager wouldn't have generated this
	return fail(Status.INTERNAL.withDescription(
	errorPrefix() + "Unable to find non-dynamic root cluster"));
	}

[ejona86]

This is caused by XdsDepManager dropping a cluster because the route no longer uses it, but XdsNR is keeping the cluster in cluster manager alive because there's still RPCs that are in the process of being routed to it.

A74 talks about this some, but I'm talking with others to understand what it means.

Note that when a cds LB policy obtains a dynamic subscription to a cluster, there may be subsequent updates from the xds resolver that do not yet include the newly subscribed-to cluster. Therefore, the cds policy will need to ignore updates in which the XdsConfig does not contain the cluster. However, this case should not occur with non-dynamic clusters, because the xds resolver ensures that those clusters are being subscribed to until they are removed from the LB policy config.

[ejona86]

We're missing this piece:

As per gRFC A31, the ConfigSelector gives each RPC a ref to the cluster that was selected for it to ensure that the cluster is not removed from the xds_cluster_manager LB policy config before the RPC is done with its LB picks. These cluster refs will also hold a subscription for the cluster from the XdsDependencyManager, so that the XdsDependencyManager will not stop watching the cluster resource until the cluster is removed from the xds_cluster_manager LB policy config.