Unknown SD when looking up NFS storage domain in startSpm

[aesteve-rh]

Description of problem

This bug was split from downstream bug ID 2048545.

When looking up a storage domain, vdsm uses the domain find method:

150     def _findDomain(self, sdUUID):
151         try:
152             findMethod = self.knownSDs[sdUUID]
153         except KeyError:
154             findMethod = self._findUnfetchedDomain
155 
156         return findMethod(sdUUID)

But if the domain is unknown, vdsm fallbacks to search the storage
domain in all connected servers:

158     def _findUnfetchedDomain(self, sdUUID):
...
171         self.log.info("Looking up domain %s", sdUUID)
172         with utils.stopwatch(
173                 "Looking up domain {}".format(sdUUID),
174                 level=logging.INFO,
175                 log=self.log):
176             for mod in (blockSD, glusterSD, localFsSD, nfsSD):
177                 try:
178                     return mod.findDomain(sdUUID)
179                 except se.StorageDomainDoesNotExist:
180                     pass
181                 except Exception:
182                     self.log.error(
183                         "Error while looking for domain `%s`",
184                         sdUUID, exc_info=True)

In this case, when looking up the storage domain in blockSD, we run lvm vgs (and fail).

Actions

_findUnfetchedDomain looks for the storage domain in all storage domain types, and it could not find it.
Why is the storage domain unknown?

Domains find methods are added to knownSDs dict when connecting
to storage server. They should not be deleted when the domain is in use.
Maybe there is a bug in this mechanism?

We should investigate this, fix the issue, and verify it.