Cloudstack agent should proactively check for changes made for Global setting parameter (host)

[kiranchavala]

ISSUE TYPE

• Improvement/Enhancement Request

COMPONENT NAME

Component:agent
Component:management-server
Component:kvm

CLOUDSTACK VERSION

4.17.2

SUMMARY

Cloudstack agent should proactively check for changes made for Global setting parameter (host)

STEPS TO REPRODUCE

1. Create a environment with multiple (2)management servers

2. The global setting parameter "host" is set to Management server 1 ip address.

3. All the hosts and systemvm point to the Management server 1 ip address

Verify it in on the host and the systemvm

cat /etc/cloudstack/agent/agent.properties
cat /usr/local/cloud/systemvm/conf/agent.properties

4. Admin user adds the second management server ip address to the global setting host

host= management server 1, management server 2

5. Admin user stops management server 1,

6. The hosts and systemvms will get into a disconnected state

7. In an ideal scneario, cloudstack agent should automataically connect to second management server ip address

Ideally cloudstack should update the agent properties file present on the host and systemvm with the 2nd management server ip and restart the agent service

to connect back to the running management server

5. Currently the hosts and systemvms remain in a disconnected state unless we perform the following steps

6. Manually edit the agent.properties in the kvm and restart the agent service

7. Destroy the systemvms so that they connect to the correct management server

EXPECTED RESULTS

Host and systemvms should connect to the second management server in case first management server is down

ACTUAL RESULTS

Host and systemvms remain in a disconnected state and they don't connect to the second management server

[kiranchavala]

This issue can solved if there is a Loadbalancer in front of the management servers

but that would be needing a additional resources for the end user

[weizhouapache]

@kiranchavala
some global configurations are not dynamic which requires a restart of management servers.

[Hudratronium]

This issue can solved if there is a Loadbalancer in front of the management servers

but that would be needing a additional resources for the end user

Which is the current suggested approach for this kind of request - https://docs.cloudstack.apache.org/en/latest/conceptsandterminology/choosing_deployment_architecture.html#multi-node-management-server

I guess i would suggest not a 'automatic' transfer but more kind of a option to 'update' hosts with new relevant setting infos. like the 'Provision host security keys' function. That could solve the issue of 'restarting' the management-server kind of scope the settings needed to transfer?

[kiranchavala]

@weizhouapache

The configuration parameter for "host" is dynamic , so there is no management service restart is required.

We have to restart the agent "cloudstack-agent " only after making the changes in the files

cat /etc/cloudstack/agent/agent.properties
cat /usr/local/cloud/systemvm/conf/agent.properties

MariaDB [cloud]> select * from configuration where name="host" \G;
*************************** 1. row ***************************
     category: Advanced
     instance: DEFAULT
    component: ApiServiceConfiguration
         name: host
        value: 10.0.32.222,10.0.33.251
  description: The ip address of management server. This can also accept comma separated addresses.
default_value: localhost
      updated: 2023-03-13 10:35:51
        scope: Global
   is_dynamic: 1
     group_id: 6
  subgroup_id: 26
       parent: NULL
 display_text: Host
         kind: CSV
      options: NULL

[weizhouapache]

I had a quick check, the new setting should be propogated to all agents.

https://github.com/apache/cloudstack/blob/main/server/src/main/java/org/apache/cloudstack/agent/lb/IndirectAgentLBServiceImpl.java#L209-L227

    @Override
    public void propagateMSListToAgents() {
        LOG.debug("Propagating management server list update to agents");
        final String lbAlgorithm = getLBAlgorithmName();
        final Map<Long, List<Long>> dcOrderedHostsMap = new HashMap<>();
        for (final Host host : getAllAgentBasedHosts()) {
            final Long dcId = host.getDataCenterId();
            if (!dcOrderedHostsMap.containsKey(dcId)) {
                dcOrderedHostsMap.put(dcId, getOrderedHostIdList(dcId));
            }
            final List<String> msList = getManagementServerList(host.getId(), host.getDataCenterId(), dcOrderedHostsMap.get(dcId));
            final Long lbCheckInterval = getLBPreferredHostCheckInterval(host.getClusterId());
            final SetupMSListCommand cmd = new SetupMSListCommand(msList, lbAlgorithm, lbCheckInterval);
            final Answer answer = agentManager.easySend(host.getId(), cmd);
            if (answer == null || !answer.getResult()) {
                LOG.warn(String.format("Failed to setup management servers list to the agent of %s", host));
            }
        }
    }

it looks like there is a bug somewhere.

@weizhouapache

The configuration parameter for "host" is dynamic , so there is no management service restart is required.

We have to restart the agent "cloudstack-agent " only after making the changes in the files

cat /etc/cloudstack/agent/agent.properties cat /usr/local/cloud/systemvm/conf/agent.properties

MariaDB [cloud]> select * from configuration where name="host" \G;
*************************** 1. row ***************************
     category: Advanced
     instance: DEFAULT
    component: ApiServiceConfiguration
         name: host
        value: 10.0.32.222,10.0.33.251
  description: The ip address of management server. This can also accept comma separated addresses.
default_value: localhost
      updated: 2023-03-13 10:35:51
        scope: Global
   is_dynamic: 1
     group_id: 6
  subgroup_id: 26
       parent: NULL
 display_text: Host
         kind: CSV
      options: NULL

[weizhouapache]

@kiranchavala
I tried to reproduce the issue

• created a env with 2 management servers
• by default, host is set to first mgmt server IP, agent.properties also uses the first mgmt IP as host

host=10.0.33.164@static

• change the global setting host to both mgmt IPs (on mgmt02 UI), the agent.properties are changed (on kvm host and system VMs)

host=10.0.33.164,10.0.34.183@static

• stop mgmt server 1, the kvm hosts and system VMs are still Up

please note, the new host is propogated to kvm hosts one by one. If a kvm host is stuck, the process might take long time.
it is supported since #3491