Prevent infinite autoscaling #11244

sudo87 · 2025-07-18T09:09:50Z

Description

This PR intends to fix issue: #9318.

New Global config is introduced, to be used as threshold for Error instances count for ASG

autoscale.errored.instance.threshold: 10 (default)

If any ASG has Error instances more than threshold, autoscaling will be stopped.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-07-18T09:17:50Z

Codecov Report

❌ Patch coverage is 23.07692% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 15.18%. Comparing base (bf46458) to head (49f898c).
⚠️ Report is 8 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...d/network/as/dao/AutoScaleVmGroupVmMapDaoImpl.java	0.00%	7 Missing ⚠️
...ava/com/cloud/network/as/AutoScaleManagerImpl.java	25.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##              4.19   #11244       +/-   ##
============================================
+ Coverage     4.28%   15.18%   +10.89%     
- Complexity       0    11363    +11363     
============================================
  Files          372     5415     +5043     
  Lines        29746   475920   +446174     
  Branches      5230    58097    +52867     
============================================
+ Hits          1274    72245    +70971     
- Misses       28327   395590   +367263     
- Partials       145     8085     +7940

Flag	Coverage Δ
uitests	`4.28% <ø> (+<0.01%)`	⬆️
unittests	`15.90% <23.07%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sudo87 · 2025-07-18T09:22:44Z

@blueorangutan package

blueorangutan · 2025-07-18T09:24:04Z

@sudo87 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-18T10:12:10Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 14255

blueorangutan · 2025-07-18T14:22:59Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14257

sudo87 · 2025-07-24T11:46:16Z

@blueorangutan test

blueorangutan · 2025-07-24T11:48:06Z

@sudo87 a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-07-25T03:19:43Z

[SF] Trillian test result (tid-13876)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52782 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11244-t13876-kvm-ol8.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

DaanHoogland

clgtm

server/src/main/java/com/cloud/network/as/AutoScaleManagerImpl.java

….java

sureshanaparti

clgtm

vishesh92 · 2025-07-26T08:20:46Z

@blueorangutan package

blueorangutan · 2025-07-26T08:22:04Z

@vishesh92 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-26T10:11:34Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14390

sureshanaparti · 2025-07-29T06:08:47Z

Verified with autoscaling with config 'autoscale.errored.instance.threshold' set to 5 (and manually set instances to Error for scaling). Instance scaling stopped once Error-ed instances reached the threshold value.

CMK:

(localcloud) 🐱 > list configurations name=autoscale.errored.instance.threshold
{
  "configuration": [
    {
      "category": "Advanced",
      "component": "AutoScaleManager",
      "defaultvalue": "10",
      "description": "The number of Error Instances allowed in autoscale vm groups for scale up.",
      "displaytext": "Autoscale errored instance threshold",
      "group": "Compute",
      "isdynamic": true,
      "name": "autoscale.errored.instance.threshold",
      "subgroup": "Virtual Machine",
      "type": "Number",
      "value": "5"
    }
  ],
  "count": 1
}

(localcloud) 🐱 > list autoscalepolicies 
{
  "autoscalepolicy": [
    {
      "account": "admin",
      "action": "SCALEDOWN",
      "conditions": [
        {
          "account": "admin",
          "counter": {
            "id": "3a416e0e-6bc4-11f0-b052-1e0073000395",
            "name": "VM CPU - average percentage",
            "provider": "VirtualRouter",
            "source": "CPU",
            "value": "vm.cpu.average.percentage"
          },
          "counterid": "3a416e0e-6bc4-11f0-b052-1e0073000395",
          "countername": "VM CPU - average percentage",
          "domain": "ROOT",
          "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
          "domainpath": "/",
          "id": "8a19ad80-dfa4-4b81-bc87-dcf95de0a083",
          "relationaloperator": "LT",
          "threshold": 2
        }
      ],
      "domain": "ROOT",
      "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
      "domainpath": "/",
      "duration": 30,
      "id": "f11ba6e7-9019-4d05-8be4-94438177f8d7",
      "name": "ScaleDownPolicy-0",
      "quiettime": 10
    },
    {
      "account": "admin",
      "action": "SCALEUP",
      "conditions": [
        {
          "account": "admin",
          "counter": {
            "id": "3a423d34-6bc4-11f0-b052-1e0073000395",
            "name": "Load Balancer - average connections per vm",
            "provider": "VirtualRouter",
            "source": "VIRTUALROUTER",
            "value": "virtual.network.lb.average.connections"
          },
          "counterid": "3a423d34-6bc4-11f0-b052-1e0073000395",
          "countername": "Load Balancer - average connections per vm",
          "domain": "ROOT",
          "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
          "domainpath": "/",
          "id": "7cb7a18a-1e17-4d3f-8646-181c26b3dc5e",
          "relationaloperator": "GT",
          "threshold": 2
        },
        {
          "account": "admin",
          "counter": {
            "id": "3a416e0e-6bc4-11f0-b052-1e0073000395",
            "name": "VM CPU - average percentage",
            "provider": "VirtualRouter",
            "source": "CPU",
            "value": "vm.cpu.average.percentage"
          },
          "counterid": "3a416e0e-6bc4-11f0-b052-1e0073000395",
          "countername": "VM CPU - average percentage",
          "domain": "ROOT",
          "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
          "domainpath": "/",
          "id": "8f2c2551-ed2f-4047-bc7f-47601f259356",
          "relationaloperator": "GT",
          "threshold": 1
        }
      ],
      "domain": "ROOT",
      "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
      "domainpath": "/",
      "duration": 30,
      "id": "2c5d2f97-186e-4243-9a93-672f74412e83",
      "name": "ScaleUpPolicy-0",
      "quiettime": 10
    }
  ],
  "count": 2
}
(localcloud) 🐱 > list autoscalevmgroups 
{
  "autoscalevmgroup": [
    {
      "account": "admin",
      "associatednetworkid": "6dac2b2f-77de-46bb-b170-16e44e567575",
      "associatednetworkname": "testvm01",
      "availablevirtualmachinecount": 1,
      "created": "2025-07-29T04:32:43+0000",
      "domain": "ROOT",
      "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
      "domainpath": "/",
      "fordisplay": true,
      "hasannotations": false,
      "id": "0750c8d2-7a85-4675-9d01-5c4deb5a31b1",
      "interval": 5,
      "lbprovider": "VirtualRouter",
      "lbruleid": "9b14260c-dfa8-43f5-84af-a629831158f7",
      "maxmembers": 10,
      "minmembers": 2,
      "name": "AutoScalingGroup01",
      "privateport": "80",
      "publicip": "10.0.55.10",
      "publicipid": "0a67c52b-e8bb-4954-b989-e389b8470b73",
      "publicport": "80",
      "scaledownpolicies": [
        {
          "account": "admin",
          "action": "SCALEDOWN",
          "conditions": [
            {
              "account": "admin",
              "counter": {
                "id": "3a416e0e-6bc4-11f0-b052-1e0073000395",
                "name": "VM CPU - average percentage",
                "provider": "VirtualRouter",
                "source": "CPU",
                "value": "vm.cpu.average.percentage"
              },
              "counterid": "3a416e0e-6bc4-11f0-b052-1e0073000395",
              "countername": "VM CPU - average percentage",
              "domain": "ROOT",
              "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
              "domainpath": "/",
              "id": "8a19ad80-dfa4-4b81-bc87-dcf95de0a083",
              "relationaloperator": "LT",
              "threshold": 2
            }
          ],
          "domain": "ROOT",
          "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
          "domainpath": "/",
          "duration": 30,
          "id": "f11ba6e7-9019-4d05-8be4-94438177f8d7",
          "name": "ScaleDownPolicy-0",
          "quiettime": 10
        }
      ],
      "scaleuppolicies": [
        {
          "account": "admin",
          "action": "SCALEUP",
          "conditions": [
            {
              "account": "admin",
              "counter": {
                "id": "3a423d34-6bc4-11f0-b052-1e0073000395",
                "name": "Load Balancer - average connections per vm",
                "provider": "VirtualRouter",
                "source": "VIRTUALROUTER",
                "value": "virtual.network.lb.average.connections"
              },
              "counterid": "3a423d34-6bc4-11f0-b052-1e0073000395",
              "countername": "Load Balancer - average connections per vm",
              "domain": "ROOT",
              "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
              "domainpath": "/",
              "id": "7cb7a18a-1e17-4d3f-8646-181c26b3dc5e",
              "relationaloperator": "GT",
              "threshold": 2
            },
            {
              "account": "admin",
              "counter": {
                "id": "3a416e0e-6bc4-11f0-b052-1e0073000395",
                "name": "VM CPU - average percentage",
                "provider": "VirtualRouter",
                "source": "CPU",
                "value": "vm.cpu.average.percentage"
              },
              "counterid": "3a416e0e-6bc4-11f0-b052-1e0073000395",
              "countername": "VM CPU - average percentage",
              "domain": "ROOT",
              "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
              "domainpath": "/",
              "id": "8f2c2551-ed2f-4047-bc7f-47601f259356",
              "relationaloperator": "GT",
              "threshold": 1
            }
          ],
          "domain": "ROOT",
          "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
          "domainpath": "/",
          "duration": 30,
          "id": "2c5d2f97-186e-4243-9a93-672f74412e83",
          "name": "ScaleUpPolicy-0",
          "quiettime": 10
        }
      ],
      "state": "ENABLED",
      "vmprofileid": "b43487ae-cf9c-404e-a58b-b99df6284256"
    }
  ],
  "count": 1
}
(localcloud) 🐱 > list autoscalevmprofiles 
{
  "autoscalevmprofile": [
    {
      "account": "admin",
      "counterparam": {},
      "domain": "ROOT",
      "domainid": "10d7a71c-6bc4-11f0-b052-1e0073000395",
      "domainpath": "/",
      "expungevmgraceperiod": 10,
      "id": "b43487ae-cf9c-404e-a58b-b99df6284256",
      "otherdeployparams": {
        "networkids": "6dac2b2f-77de-46bb-b170-16e44e567575"
      },
      "serviceofferingid": "2fd594f2-6f8e-4ceb-83eb-a92488b65dd1",
      "templateid": "10defc12-6bc4-11f0-b052-1e0073000395",
      "userdata": "IyEvYmluL2Jhc2gKeXVtIGluc3RhbGwgLXkgaHR0cGQKc2VkIC1pIC1lIOKAmC9PcHRpb25zIEluZGV4ZXMgRm9sbG93U3ltTGlua3Mvcy8kLyBFeGVjQ0dJL+KAmSBcCiAgICAgICAgICAtZSDigJhzL0RpcmVjdG9yeUluZGV4IGluZGV4Lmh0bWwvRGlyZWN0b3J5SW5kZXggaW5kZXgucHkv4oCZIFwKICAgICAgICAgIC1lIOKAmHMvI0FkZEhhbmRsZXIgY2dpLXNjcmlwdCAuY2dpL0FkZEhhbmRsZXIgY2dpLXNjcmlwdCAucHkv4oCZIC9ldGMvaHR0cGQvY29uZi9odHRwZC5jb25mCmNhdCA8PCBFT0YgPiAvdmFyL3d3dy9odG1sL2luZGV4LnB5CiMhL3Vzci9iaW4vZW52IHB5dGhvbgppbXBvcnQgdGltZQppbXBvcnQgc29ja2V0CnRpbWUuc2xlZXAoNSkKcHJpbnQo4oCYQ29udGVudC10eXBlOiB0ZXh0L2h0bWxcblxu4oCZKQpwcmludCjigJg8cCBzdHlsZT3igJ10ZXh0LWFsaWduOiBjZW50ZXI74oCdPkhlbGxvIFdvcmxkISEhPC9wPuKAmSkKcHJpbnQo4oCYPHAgc3R5bGU94oCddGV4dC1hbGlnbjogY2VudGVyO+KAnT48c3Ryb25nPiBJbnN0YW5jZTogPC9zdHJvbmc+e308L3A+4oCZLmZvcm1hdChzb2NrZXQuZ2V0aG9zdG5hbWUoKSkpCkVPRgpjaG1vZCA3MDUgL3Zhci93d3cvaHRtbC9pbmRleC5weQpzeXN0ZW1jdGwgZW5hYmxlIGh0dHBkLnNlcnZpY2UKc3lzdGVtY3RsIHJlc3RhcnQgaHR0cGQuc2VydmljZQ==",
      "zoneid": "75958f7a-8791-40ec-9992-622f7a62f2a0"
    }
  ],
  "count": 1
}

DB:

mysql> SELECT * FROM cloud.autoscale_vmgroups\G;
*************************** 1. row ***************************
              id: 1
            uuid: 0750c8d2-7a85-4675-9d01-5c4deb5a31b1
         zone_id: 1
       domain_id: 1
      account_id: 2
load_balancer_id: 1
            name: AutoScalingGroup01
     min_members: 2
     max_members: 10
     member_port: 80
        interval: 5
      profile_id: 1
           state: ENABLED
         created: 2025-07-29 04:32:43
         removed: NULL
         display: 1
   last_interval: 2025-07-29 04:40:13
     next_vm_seq: 4
1 row in set (0.00 sec)

mysql> SELECT id, uuid, name, state FROM cloud.vm_instance WHERE type = 'User' AND removed IS NULL;
+----+--------------------------------------+-----------------------------------------+---------+
| id | uuid                                 | name                                    | state   |
+----+--------------------------------------+-----------------------------------------+---------+
|  5 | c7c8a7bc-2830-43b0-bb98-9b5d2645dc5f | autoScaleVm-AutoScalingGroup01-1-veufwt | Error   |
|  6 | 7d69f64d-e28a-409f-8bc3-d159f9c7a733 | autoScaleVm-AutoScalingGroup01-2-jeptjg | Error   |
|  7 | fc518dc9-2242-4ff5-896d-b39d49c5d2a2 | autoScaleVm-AutoScalingGroup01-3-gqhjel | Error   |
|  8 | b3c8fa3e-6605-434b-8100-14e18da5454d | autoScaleVm-AutoScalingGroup01-4-tjauuh | Error   |
|  9 | 024a10ac-811d-4cc7-b22a-75320bcf8bea | autoScaleVm-AutoScalingGroup01-5-qfqqrd | Error   |
| 10 | 017def07-505c-49c9-a270-eed42bd1cc96 | autoScaleVm-AutoScalingGroup01-6-fnwfbc | Error   |
| 11 | ebc6359b-7c84-4fa7-9e26-499d5b205968 | autoScaleVm-AutoScalingGroup01-7-pjzfpi | Running |
+----+--------------------------------------+-----------------------------------------+---------+
7 rows in set (0.00 sec)

mysql> SELECT * FROM cloud.autoscale_vmgroup_vm_map;
+----+------------+-------------+
| id | vmgroup_id | instance_id |
+----+------------+-------------+
|  1 |          1 |           5 |
|  2 |          1 |           6 |
|  3 |          1 |           7 |
|  4 |          1 |           8 |
|  5 |          1 |           9 |
|  6 |          1 |          10 |
|  7 |          1 |          11 |
+----+------------+-------------+
7 rows in set (0.00 sec)

MS Logs:

2025-07-29 04:50:08,060 DEBUG [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-d3e64a24) (logid:50a61812) Start monitoring on AutoScale VmGroup AutoScaleVmGroupVO[id=1|name=AutoScalingGroup01|loadBalancerId=1|profileId=1]
2025-07-29 04:50:08,065 DEBUG [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-d3e64a24) (logid:50a61812) There are currently 1 available VMs which is less than the minimum member of the AS group (2), scaling up 1 VMs
2025-07-29 04:50:08,070 WARN  [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-d3e64a24) (logid:50a61812) Number of Errored Instances are greater than the threshold in this group for scaling up, so do nothing more
...
2025-07-29 04:50:13,074 DEBUG [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-f296c98c) (logid:97a906d8) Start monitoring on AutoScale VmGroup AutoScaleVmGroupVO[id=1|name=AutoScalingGroup01|loadBalancerId=1|profileId=1]
2025-07-29 04:50:13,079 DEBUG [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-f296c98c) (logid:97a906d8) There are currently 1 available VMs which is less than the minimum member of the AS group (2), scaling up 1 VMs
2025-07-29 04:50:13,082 WARN  [c.c.n.a.AutoScaleManagerImpl] (VmGroup-Monitor-1-1:ctx-f296c98c) (logid:97a906d8) Number of Errored Instances are greater than the threshold in this group for scaling up, so do nothing more
...

GUI/Screenshots:

Prevent infinite autoscaling

df06d8a

boring-cyborg bot added the component:networking label Jul 18, 2025

sudo87 requested review from weizhouapache, Pearl1594 and shwstppr July 18, 2025 09:09

DaanHoogland added this to the 4.19.4 milestone Jul 25, 2025

DaanHoogland approved these changes Jul 25, 2025

View reviewed changes

sureshanaparti reviewed Jul 25, 2025

View reviewed changes

server/src/main/java/com/cloud/network/as/AutoScaleManagerImpl.java Outdated Show resolved Hide resolved

Update server/src/main/java/com/cloud/network/as/AutoScaleManagerImpl…

49f898c

….java

sureshanaparti approved these changes Jul 25, 2025

View reviewed changes

sureshanaparti added the status:needs-testing label Jul 25, 2025

sureshanaparti linked an issue Jul 25, 2025 that may be closed by this pull request

Autoscale Group creates Infinite Number Of VMs when unable to startup #9318

Closed

Pearl1594 mentioned this pull request Jul 25, 2025

Prevent infinite retries of autoscaling #9574

Closed

14 tasks

sureshanaparti self-assigned this Jul 28, 2025

sureshanaparti removed the status:needs-testing label Jul 29, 2025

sureshanaparti merged commit a32738c into apache:4.19 Jul 29, 2025
26 checks passed

sureshanaparti mentioned this pull request Jul 29, 2025

Autoscale Group creates Infinite Number Of VMs when unable to startup #9318

Closed

DaanHoogland deleted the fixAutoScaleVM branch July 29, 2025 07:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Prevent infinite autoscaling #11244

Prevent infinite autoscaling #11244

Uh oh!

sudo87 commented Jul 18, 2025 •

edited

Loading

Uh oh!

codecov bot commented Jul 18, 2025 •

edited

Loading

Uh oh!

sudo87 commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

sudo87 commented Jul 24, 2025

Uh oh!

blueorangutan commented Jul 24, 2025

Uh oh!

blueorangutan commented Jul 25, 2025

Uh oh!

DaanHoogland left a comment

Uh oh!

Uh oh!

sureshanaparti left a comment

Uh oh!

vishesh92 commented Jul 26, 2025

Uh oh!

blueorangutan commented Jul 26, 2025

Uh oh!

blueorangutan commented Jul 26, 2025

Uh oh!

sureshanaparti commented Jul 29, 2025

Uh oh!

Uh oh!

Uh oh!

Prevent infinite autoscaling #11244

Prevent infinite autoscaling #11244

Uh oh!

Conversation

sudo87 commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sudo87 commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

blueorangutan commented Jul 18, 2025

Uh oh!

sudo87 commented Jul 24, 2025

Uh oh!

blueorangutan commented Jul 24, 2025

Uh oh!

blueorangutan commented Jul 25, 2025

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sureshanaparti left a comment

Choose a reason for hiding this comment

Uh oh!

vishesh92 commented Jul 26, 2025

Uh oh!

blueorangutan commented Jul 26, 2025

Uh oh!

blueorangutan commented Jul 26, 2025

Uh oh!

sureshanaparti commented Jul 29, 2025

Uh oh!

Uh oh!

Uh oh!

sudo87 commented Jul 18, 2025 •

edited

Loading

codecov bot commented Jul 18, 2025 •

edited

Loading