core: handle closed transport factory in InternalSubchannel without panic

[kannanjgithub]

Catch IllegalStateException when starting a new transport against a closed transport factory and shut down the subchannel safely instead of letting it escape into the SynchronizationContext and causing a channel panic.

Internal yaqs
Related CL

[ejona86]

When #12985 goes in, this won't be necessary, right?

[kannanjgithub]

#12985 will fix the reported problem of the transport factory lifecycle problem - to avoid closing it inadvertently when the OOB channel is shutdown by the Rls LB. However it also exposed a race condition that exists during legitimate closure of the transport factory during a channel shutdown, where an in-flight name resolution creates a new subchannel and tries to calls start a new transport.

[ejona86]

This is a hack, and is at best a temporary workaround. We should not be catching specific IllegalStateException/NullPointerException and other programmer errors; we should prevent them instead. In this case, it also assumes that any IllegalStateException is this specific exception that we have seen, but that is not guaranteed.

I don't see a log/exception that shows a NameResolver race is happening. I see one where a name resolver calls into the synchronization context, but the runnable being executed was not from the NameResolver. Any function lower than SynchronizationContext.execute() on the stack is not necessarily related to the code running within the syncContext.

We shouldn't really have problems here due to NameResolvers, because name resolvers don't create/manage subchannels. And we stop delivering updates from a NR once it is shut down.