cks: k8s cluster continue to show in scaling state even after failure #7920
Description
ISSUE TYPE
- Bug Report
COMPONENT NAME
CKS
CLOUDSTACK VERSION
4.17.2, 4.18 branch
CONFIGURATION
Advanced networking with VMware
OS / ENVIRONMENT
N/A
SUMMARY
Dynamic scaling for the cluster node fails but the cluster continues to stay in the scaling state
STEPS TO REPRODUCE
- Deploy a k8s cluster with desired size and 2vcpu, 2GB memory offering
- Scale a running cluster to a new offering with 2vcpu, 4GB memory offering. Observe dynamic scaling fails
- k8s cluster continues to show in Scaling state
EXPECTED RESULTS
k8s cluster should go in Alert state on scaling failure
ACTUAL RESULTS
k8s cluster continues to show in Scaling state until it is stopped
k8s node scaling failure which is okay by itself,
2023-08-29 08:48:30,446 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-222:ctx-04b6a466 10.0.34.175, job-157/job-158, cmd: ScaleVmCommand) (logid:df0c9af7) ScaleVmCommand failed due to: [Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Memory of VM i-2-13-VM cannot be scaled to 4096MB. Requested memory limit is beyond the hotadd memory limit for this VM at the moment is 2048MB.
].
com.cloud.utils.exception.CloudRuntimeException: Memory of VM i-2-13-VM cannot be scaled to 4096MB. Requested memory limit is beyond the hotadd memory limit for this VM at the moment is 2048MB.
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:1938)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:590)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-08-29 08:48:30,447 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-222:ctx-04b6a466) (logid:df0c9af7) Seq 1-8033577310298833327: Response Received:
2023-08-29 08:48:30,448 DEBUG [c.c.a.t.Request] (DirectAgent-222:ctx-04b6a466) (logid:df0c9af7) Seq 1-8033577310298833327: Processing: { Ans: , MgmtId: 32988721053949, via: 1(10.0.34.175), Ver: v1, Flags: 110, [{"com.cloud.agent.api.ScaleVmAnswer":{"result":"false","details":"Unable to execute ScaleVmCommand due to: [com.cloud.utils.exception.CloudRuntimeException: Memory of VM i-2-13-VM cannot be scaled to 4096MB. Requested memory limit is beyond the hotadd memory limit for this VM at the moment is 2048MB.].","wait":"0","bypassHostMaintenance":"false"}}] }
Cluster continues in scaling state,