[VMware / PVLAN] Port groups seem never cleaned up on ESXi when networks are deleted and thus block reusing the same VLANs (

[andrijapanicsb]

• Bug Report

COMPONENT NAME

Networking handling

CLOUDSTACK VERSION

4.11+ (the issue is there probably from the very first days of  VMware support)

Explanation of the problem with the steps to reproduce it:

1. create a Shared network with i.e. VLAN 900, use it, delete it (spin a VM in the network, delete the VM, delete the network) - and the port group created on ESXi host stays (as it seems) forever and is never removed (at least not inside the time frame expected/set by network.gc.wait and network.gc.interval).

On ESXi hosts, the port group with VLAN 900 is left existing forever.

2. Now, create L2 network (as opposed to Shared created previously) with the same VLAN 900 - and ACS will silently reuse the same port group that already exists on vSphere- so far so good. I assume its an expected behaviour since it seems that a (single VLAN i.e. not PVLAN) port group has the same name when created for both Shared or L2 network - so we can silently reuse the "garbage" port group without recreating it from scratch.

3. PROBLEM: if you want to use the same VLAN 900 in the PVLAN setup (on shared or L2, irrelevant) - ACS will try to create a NEW port group (due to different naming scheme for the port group) by using the same Primary VLAN 900 (and some other secondary VLAN as defined by PVLAN setup) but will fail to use VLAN 900 since it’s “used” by another port group (even though we have previously deleted such networks in ACS).

I’m testing with dvSwitches (required for PVLAN setup), if this info is of any relevance.

In the related PVLAN work ( #3732 ) we are going to make checks which will simply block the createNetwork API when any of the VLANs defined for the PVLAN setup (primary or secondary VLANs) is already used on another ACS network.
But this doesn't solve the issue on "mature" environments, where a customer will upgrade ACS, start using PVLANs, but he already has garbage port groups left from previous months/years of his ACS infra existence.

[rohityadavcloud]

@andrijapanicsb could it be possible that this bug has become a feature and some people would expect the portgroups to be there on the host(s)? cc @DaanHoogland @borisstoyanov @vladimirpetrov @alexandremattioli

[andrijapanicsb]

The problem is when the network shutdown happens (i.e. on next start of ANY VM in that network - ACS would assign a NEW/DIFFERENT VLAN) - thus the old one is removed in ACS (free to be reused) but ACS doesn't clean the port groups on VMware side.

I can't think of any case where the port group should be left - if the vlan has been "removed" (network shutdown) in ACS - it makes perfect sense that is should be removed on VMware side as well. The only case I can think is if someone is attaching some VM out-of-band, but that is not expected nor supported...

[DaanHoogland]

sounds like we want to be able to do two things:

1. autocleanup when a network is deleted/brought down
2. preserve/reserve a port group for use out of band
   I don't want to go into the argument of bug/feature but both use cases seem reasonable so let's say it is a bugfeature and plan a setting that drives this for 4.16 (which is by now pretty full)
   btw, I set this to normal complexity but the garbage collection part may have some extensive checking requirements.

[alexandremattioli]

In the case of SVS I'd leave the networks there (the vlans in ESX). There's no PVLAN conflict issue and it speeds up deployment as it can take quite a while to create all the portgroups in all hosts.

[alexandremattioli]

@rhtyd I've personally used this bug as a feature in the past.

[andrijapanicsb]

In the case of SVS I'd leave the networks there (the vlans in ESX). There's no PVLAN conflict issue and it speeds up deployment as it can take quite a while to create all the portgroups in all hosts.

AFAIK, this is (unfortunately) not true, as we always reconfigure port groups (without checking if they exist or not) - part of the ensuring things are always in place (same thing as the fact that we tear-down VM's NICs and re-attach them during VM being started).

As I communicated to Daan - if a network "shutdown" happens in CLoudStack - another VLAN will (most probably) be used for that network (when the next time a very first VM is started in that network) - so IF we are shutting down a network, let's clear our garbage as well (on hypervisor side), otherwise, if the networks are created as "persistent" - they will never be shutdown, thus all the port groups stay where they are - but this needs to be covered/implemented (persistency) for all kinds of networks (shared, isolated, VPC isolated, L2 - private gateways to be considered as well)

Your use case @alexandremattioli was to probably attach other stuff to the port groups, right? (can be achieved by ensuring networks are "persistent" and thus never go to "shutdown")

[alexandremattioli]

In my use case we'd pre-create all port groups in all ESX hosts (via PowerCLI), knowing that if the portgroup already existed ACS wouldn't need to recreate and wouldn't destroy them if the network is deleted. Without that the first VM of a new network would take 30-60 minutes to be deployed, with the portgroup in place this went down to 5-10 minutes. I do agree with the garbage collection, but as @rhtyd pointed out it's very likely that more customers started using this bug as a feature. An option would be to have a global setting like vmware.network.autocleanup.

[andrijapanicsb]

+1 on the global setting, yes.