Unable to configure higher threshold for public IP usage

[KuasarCloud]

Hi team,

I’ve reached 85% usage of my public IPs, but I haven’t been able to find an option in the Global Configuration to adjust the threshold. Ideally, I’d like to set it to at least 90%.

Is there a way to configure this, or is the threshold value fixed?

Thanks in advance!

[prashanthr2]

@KuasarCloud Below global setting parameters should address your issue (may require management server restart for the changes to take effect)

zone.virtualnetwork.publicip.capacity.notificationthreshold
zone.directnetwork.publicip.capacity.notificationthreshold

[prashanthr2]

@KuasarCloud As per your last response you do have free IP's. However, the log suggests it cannot find a public IP during VPC creation.

Can you please send the full management server log we will need to go through the complete log/job to get to the bottom of this.

Just a FYI, AFAIK there is no setting or config in cloudstack that will restrict assignments of public IP's once it hits a threshold. Looks like something else is happening here.

[KuasarCloud]

Hi @prashanthr2, this is complete logid

sudo cat /var/log/cloudstack/management/management-server.log |grep 1a42b6a5 2025-10-01 16:59:59,595 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp1390913202-24:[ctx-96b0833e, ctx-8bbd79a2]) (logid:fb8d0988) submit async job-53744, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin","cmdInfo":"{\"sessionkey\":\"tSWxOjqN17FRLnqqhJNNmf_P7E4\",\"start\":\"true\",\"httpmethod\":\"GET\",\"ctxAccountId\":\"2\",\"uuid\":\"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b\",\"cmdEventType\":\"VPC.CREATE\",\"vpcofferingid\":\"58f2562a-5efc-4db1-8adc-c3dae8a794a3\",\"displaytext\":\"test\",\"response\":\"json\",\"ctxUserId\":\"2\",\"name\":\"test\",\"zoneid\":\"d8b68128-960f-474e-9902-5fb7b9370d7b\",\"cidr\":\"10.10.0.0\/16\",\"ctxStartEventId\":\"998320\",\"id\":\"194\",\"ctxDetails\":\"{\\\"interface com.cloud.network.vpc.Vpc\\\":\\\"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b\\\",\\\"interface com.cloud.dc.DataCenter\\\":\\\"d8b68128-960f-474e-9902-5fb7b9370d7b\\\",\\\"interface com.cloud.network.vpc.VpcOffering\\\":\\\"58f2562a-5efc-4db1-8adc-c3dae8a794a3\\\",\\\"interface com.cloud.projects.Project\\\":\\\"f8d476da-9d78-47e0-be78-996f434a5328\\\"}\",\"projectid\":\"f8d476da-9d78-47e0-be78-996f434a5328\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":53744,"initMsid":2199211933721,"instanceId":194,"instanceType":"None","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"1a42b6a5-c643-470c-93d5-d4d569b69796"} 2025-10-01 16:59:59,596 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin","cmdInfo":"{\"sessionkey\":\"tSWxOjqN17FRLnqqhJNNmf_P7E4\",\"start\":\"true\",\"httpmethod\":\"GET\",\"ctxAccountId\":\"2\",\"uuid\":\"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b\",\"cmdEventType\":\"VPC.CREATE\",\"vpcofferingid\":\"58f2562a-5efc-4db1-8adc-c3dae8a794a3\",\"displaytext\":\"test\",\"response\":\"json\",\"ctxUserId\":\"2\",\"name\":\"test\",\"zoneid\":\"d8b68128-960f-474e-9902-5fb7b9370d7b\",\"cidr\":\"10.10.0.0\/16\",\"ctxStartEventId\":\"998320\",\"id\":\"194\",\"ctxDetails\":\"{\\\"interface com.cloud.network.vpc.Vpc\\\":\\\"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b\\\",\\\"interface com.cloud.dc.DataCenter\\\":\\\"d8b68128-960f-474e-9902-5fb7b9370d7b\\\",\\\"interface com.cloud.network.vpc.VpcOffering\\\":\\\"58f2562a-5efc-4db1-8adc-c3dae8a794a3\\\",\\\"interface com.cloud.projects.Project\\\":\\\"f8d476da-9d78-47e0-be78-996f434a5328\\\"}\",\"projectid\":\"f8d476da-9d78-47e0-be78-996f434a5328\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":53744,"initMsid":2199211933721,"instanceId":194,"instanceType":"None","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"1a42b6a5-c643-470c-93d5-d4d569b69796"} 2025-10-01 16:59:59,637 DEBUG [c.c.a.ApiServlet] (qtp1390913202-23:[ctx-86a69b38]) (logid:1c2ef1ed) ===START=== 10.1.2.2 -- GET jobId=1a42b6a5-c643-470c-93d5-d4d569b69796&command=queryAsyncJobResult&response=json&sessionkey=tSWxOjqN17FRLnqqhJNNmf_P7E4&projectid=f8d476da-9d78-47e0-be78-996f434a5328 2025-10-01 16:59:59,671 DEBUG [o.a.c.n.r.d.VpcRouterDeploymentDefinition] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Verifying router service offering with uuid : 7f3b1a33-a562-4520-b2d1-70836d964b18 2025-10-01 16:59:59,675 DEBUG [o.a.c.n.r.d.VpcRouterDeploymentDefinition] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Service offering Service offering {"id":60,"name":"Kuasar Basic Virtual Router Offering","uuid":"7f3b1a33-a562-4520-b2d1-70836d964b18"}. will be used on virtual router 2025-10-01 16:59:59,680 DEBUG [c.c.a.ApiServlet] (qtp1390913202-23:[ctx-86a69b38, ctx-de980158]) (logid:1c2ef1ed) ===END=== 10.1.2.2 -- GET jobId=1a42b6a5-c643-470c-93d5-d4d569b69796&command=queryAsyncJobResult&response=json&sessionkey=tSWxOjqN17FRLnqqhJNNmf_P7E4&projectid=f8d476da-9d78-47e0-be78-996f434a5328 2025-10-01 16:59:59,685 DEBUG [c.c.n.IpAddressManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) lock account Account [{"accountName":"PrjAcct-61b0cf2797f7a27e92298e45-2","id":31,"uuid":"ba450bc1-8d8c-45e8-a5f3-60c2a955f375"}] is acquired 2025-10-01 16:59:59,705 WARN [c.c.n.IpAddressManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Unable to get ip address in zone id=2, vlanId id=[4, 6], vlanId id=[2] 2025-10-01 16:59:59,707 DEBUG [c.c.u.d.T.Transaction] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Rolling back the transaction: Time = 26 Name = API-Job-Executor-72; called by -TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-TransactionLegacy.close:663-Transaction.execute:36-IpAddressManagerImpl.assignDedicateIpAddress:1123-VpcManagerImpl.assignSourceNatIpAddressToVpc:3357-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:77-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:569-AopUtils.invokeJoinpointUsingReflection:344-ReflectiveMethodInvocation.invokeJoinpoint:198 2025-10-01 16:59:59,708 DEBUG [c.c.n.IpAddressManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Releasing lock account Account [{"accountName":"PrjAcct-61b0cf2797f7a27e92298e45-2","id":31,"uuid":"ba450bc1-8d8c-45e8-a5f3-60c2a955f375"}] 2025-10-01 16:59:59,713 ERROR [c.c.n.IpAddressManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Unable to get source nat ip address for account Account [{"accountName":"PrjAcct-61b0cf2797f7a27e92298e45-2","id":31,"uuid":"ba450bc1-8d8c-45e8-a5f3-60c2a955f375"}] 2025-10-01 16:59:59,719 DEBUG [o.a.c.n.r.d.VpcRouterDeploymentDefinition] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Lock is released for vpc [id: 194] (VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"}) as a part of router startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(2)-Pod(null)-Cluster(null)-Host(null)-Storage()] 2025-10-01 16:59:59,719 WARN [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Failed to start vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} due to com.cloud.exception.InsufficientAddressCapacityException: Insufficient address capacityScope=interface com.cloud.dc.DataCenter; id=2 2025-10-01 16:59:59,720 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Destroying vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} that failed to start 2025-10-01 16:59:59,720 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Destroying vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} 2025-10-01 16:59:59,723 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Updating VPC VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} with state Inactive as a part of vpc delete 2025-10-01 16:59:59,726 DEBUG [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Updating resource Type = vpc count for Account = 31 Operation = decreasing Amount = 1 2025-10-01 16:59:59,739 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Shutting down vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} 2025-10-01 16:59:59,748 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} has been shutdown successfully 2025-10-01 16:59:59,749 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Cleaning up resources for vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} 2025-10-01 16:59:59,749 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Cleaning up existed site to site VPN connections 2025-10-01 16:59:59,750 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Cleaning up existed site to site VPN gateways 2025-10-01 16:59:59,753 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Releasing ips for vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} as a part of vpc cleanup 2025-10-01 16:59:59,753 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Released ip addresses for vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} as a part of cleanup vpc process 2025-10-01 16:59:59,755 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Found 0 to revoke for the vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} 2025-10-01 16:59:59,774 DEBUG [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} is destroyed successfully 2025-10-01 16:59:59,783 WARN [c.c.n.v.VpcManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Successfully destroyed vpc VPC {"id":194,"name":"test","uuid":"993ab48f-cf1e-42a1-9ed5-2ac5d3c5928b"} that failed to start 2025-10-01 16:59:59,783 WARN [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Skipping delivering event Event {"description":"{\"event\":\"VPC.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"VPC.CREATE","eventUuid":null,"resourceType":"Vpc","resourceUUID":null} to any webhook as account ID is missing 2025-10-01 16:59:59,783 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744, ctx-1cf995dd]) (logid:1a42b6a5) Failed to publish event [category: ActionEvent, type: VPC.CREATE] on bus webhookEventBus 2025-10-01 16:59:59,808 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Complete async job-53744, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to create VPC"} 2025-10-01 16:59:59,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Publish async job-53744 complete on message bus 2025-10-01 16:59:59,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Wake up jobs related to job-53744 2025-10-01 16:59:59,810 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Update db status for job-53744 2025-10-01 16:59:59,812 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Wake up jobs joined with job-53744 and disjoin all subjobs created from job- 53744 2025-10-01 16:59:59,823 DEBUG [c.c.a.ApiServer] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Retrieved cmdEventType from job info: VPC.CREATE 2025-10-01 16:59:59,829 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Done executing org.apache.cloudstack.api.command.admin.vpc.CreateVPCCmdByAdmin for job-53744 2025-10-01 16:59:59,829 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-72:[ctx-3827081d, job-53744]) (logid:1a42b6a5) Remove job-53744 from job monitoring 2025-10-01 17:00:02,775 DEBUG [c.c.a.ApiServlet] (qtp1390913202-23:[ctx-0d89fb03]) (logid:010da591) ===START=== 10.1.2.2 -- GET jobId=1a42b6a5-c643-470c-93d5-d4d569b69796&command=queryAsyncJobResult&response=json&sessionkey=tSWxOjqN17FRLnqqhJNNmf_P7E4&projectid=f8d476da-9d78-47e0-be78-996f434a5328 2025-10-01 17:00:02,854 DEBUG [c.c.a.ApiServlet] (qtp1390913202-23:[ctx-0d89fb03, ctx-543447b2]) (logid:010da591) ===END=== 10.1.2.2 -- GET jobId=1a42b6a5-c643-470c-93d5-d4d569b69796&command=queryAsyncJobResult&response=json&sessionkey=tSWxOjqN17FRLnqqhJNNmf_P7E4&projectid=f8d476da-9d78-47e0-be78-996f434a5328

[KuasarCloud]

I think it's because the free IP are for the System Pool, not usable for users VPCs

[prashanthr2]

@KuasarCloud All the public IP's are maintained user_ip_address table and as per the query output you shared there are free IP's.

>>I think it's because the free IP are for the System Pool,

This could be the issue.. Did you dedicate or reserve public IP's for systemVM's? Can you share the output of below queries

select count() from user_ip_address where allocated is null and state='Free' and data_center_id=2 and forsystemvms=1;
select count() from user_ip_address where allocated is null and state='Free' and data_center_id=2 and forsystemvms=0;
select count(*) from user_ip_address where allocated is null and state='Free' and data_center_id=2;

If query 2 returns count as 0 then you have no public IP's for user networks consumption. You will need to add more IP's.