Usage server not working #10914

TadiosAbebe · 2025-05-24T09:05:57Z

TadiosAbebe
May 24, 2025

Hello everyone,
I can't seem to figure out the following issue i have with the usage server. I remember it was working before but now the usage.err and usage.out logs are throwing Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist. I did not know what i mess up.

usage.err

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "main" java.lang.RuntimeException: Unable to start com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_480aaa40
        at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:155)
        at com.cloud.usage.UsageServer.start(UsageServer.java:55)
        at com.cloud.usage.UsageServer.main(UsageServer.java:42)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:441)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$51(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>)
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$50(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>)
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$18(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>)
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$16(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl.getLastJob(UsageJobDaoImpl.java:150)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$getLastJob$5(<generated>)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>)
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123)
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.getLastJob(<generated>)
        at com.cloud.usage.UsageManagerImpl.start(UsageManagerImpl.java:344)
        at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:149)
        ... 2 more

usage.out

08:43:49.671 [Usage-HB-1] ERROR com.cloud.usage.UsageManagerImpl.Heartbeat - error in heartbeat
com.cloud.utils.exception.CloudRuntimeException: Unable to find on DB, due to: (conn=122596) Table 'cloud.usage_job' doesn't exist
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:441) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$51(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?]
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$searchIncludingRemoved$50(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?]
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.searchIncludingRemoved(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$18(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?]
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463) ~[cloud-framework-db-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$search$16(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?]
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.search(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl.getNextImmediateJob(UsageJobDaoImpl.java:178) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.CGLIB$getNextImmediateJob$6(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628_FastClassByCloudStack_e40b44cf.invoke(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228) ~[cglib-nodep-3.3.0.jar:?]
        at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:123) ~[cloud-utils-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628.getNextImmediateJob(<generated>) ~[cloud-engine-schema-4.20.0.0.jar:4.20.0.0]
        at com.cloud.usage.UsageManagerImpl$Heartbeat.runInContext(UsageManagerImpl.java:2192) ~[cloudstack-usage.jar:4.20.0.0]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0]
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0]
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) ~[cloud-framework-managed-context-4.20.0.0.jar:4.20.0.0]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:840) ~[?:?]
08:44:49.659 [Usage-HB-1] ERROR com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_a6924628 - DB Exception on: HikariProxyPreparedStatement@438883728 wrapping ServerPreparedStatement{sql:'SELECT usage_job.id, usage_job.host, usage>
java.sql.SQLSyntaxErrorException: (conn=122596) Table 'cloud.usage_job' doesn't exist
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:282) ~[mariadb-java-client-3.1.4.jar:?]
        at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:370) ~[mariadb-java-client-3.1.4.jar:?]
        at org.mariadb.jdbc.message.client.PrepareExecutePacket.readPacket(PrepareExecutePacket.java:153) ~[mariadb-java-client-3.1.4.jar:?]

I have tried to remove and re-install the cloudstack-usage service but the issue is still the same

The management server log also have similar log every few minutes

management-server.log

2025-05-24 05:22:12,322 ERROR [c.c.u.d.UsageJobDaoImpl] (HA-3:[ctx-7cfd0587]) (logid:585b600e) DB Exception on: HikariProxyPreparedStatement@1302901535 wrapping ServerPreparedStatement{sql:'SELECT usage_job.id, usage_job.host, usage_job.pid, usage_job.job_type, usage_job.scheduled, usage_job.start_millis, usage_job.end_millis, usage_job.exec_time, usage_job.start_date, usage_job.end_date, usage_job.success, usage_job.heartbeat FROM usage_job  ORDER BY usage_job.heartbeat DESC  LIMIT 0, 1', parameters:[]} java.sql.SQLSyntaxErrorException: (conn=121197) Table 'cloud.usage_job' doesn't exist
        at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:282)
        at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:370)
        at org.mariadb.jdbc.message.client.PrepareExecutePacket.readPacket(PrepareExecutePacket.java:153)
        at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:872)
        at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:811)
        at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:670)
        at org.mariadb.jdbc.client.impl.MultiPrimaryClient.execute(MultiPrimaryClient.java:347)
        at org.mariadb.jdbc.ServerPreparedStatement.executePipeline(ServerPreparedStatement.java:117)
        at org.mariadb.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:95)
        at org.mariadb.jdbc.ServerPreparedStatement.executeQuery(ServerPreparedStatement.java:431)
        at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
        at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:434)
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:368)
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:357)
        at com.cloud.utils.db.GenericDaoBase.search(GenericDaoBase.java:1463)
        at com.cloud.usage.dao.UsageJobDaoImpl.getLastHeartbeat(UsageJobDaoImpl.java:190)
        at jdk.internal.reflect.GeneratedMethodAccessor382.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy302.getLastHeartbeat(Unknown Source)
        at com.cloud.ha.HighAvailabilityManagerExtImpl$UsageServerMonitorTask.runInContext(HighAvailabilityManagerExtImpl.java:73)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecuto

Anyone has idea on how i can resolve the above issue or why it happened ? I am using cloudstack 4.20.

Answered by winterhazel

May 24, 2025

Both Usage and the MS are trying to find Usage's tables in the cloud database instead of cloud_usage.

What value does the property db.usage.name have in your /etc/cloudstack/management/db.properties? It should be set to cloud_usage.

View full answer

winterhazel · 2025-05-24T18:57:01Z

winterhazel
May 24, 2025
Collaborator

Both Usage and the MS are trying to find Usage's tables in the cloud database instead of cloud_usage.

What value does the property db.usage.name have in your /etc/cloudstack/management/db.properties? It should be set to cloud_usage.

3 replies

TadiosAbebe May 24, 2025
Author

@winterhazel , thank you for your reply

It is cloud_usage

root@zcs01mgtvm01:/var/log/cloudstack/usage# grep db.usage.name /etc/cloudstack/management/db.properties
db.usage.name=cloud_usage

root@zcs01mgtvm01:/var/log/cloudstack/usage# ll /etc/cloudstack/usage/
total 8
drwxr-xr-x 2 root root 4096 May 24 11:43 ./
drwxr-xr-x 4 root root 4096 May 24 11:43 ../
lrwxrwxrwx 1 root root   40 May 24 11:43 db.properties -> /etc/cloudstack/management/db.properties
lrwxrwxrwx 1 root root   30 May 24 11:43 key -> /etc/cloudstack/management/key

TadiosAbebe May 25, 2025
Author

I managed to find the issue, it was my bad. I overlooked the database name in db.usage.url where i copied the value from db.cloud.url which was pointing to the cloud database instead of cloud_usage after changing the database name in the db.usage.url usage server started working. @winterhazel Thank you for pointing me in the right direction.

db.usage.uri=jdbc:mariadb:sequential://10.0.6.31:3306,10.0.6.32:3306,10.0.6.33:3306/cloud_usage?autoReconnect=true&prepStmtCacheSize=517&cachePrepStmts=true&sessionVariables=sql_mode='STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION'&serverTimezone=UTC

winterhazel May 25, 2025
Collaborator

@TadiosAbebe great, thanks for returning what the issue was.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Usage server not working #10914

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Usage server not working #10914

Uh oh!

TadiosAbebe May 24, 2025

Replies: 1 comment · 3 replies

Uh oh!

winterhazel May 24, 2025 Collaborator

Uh oh!

TadiosAbebe May 24, 2025 Author

Uh oh!

TadiosAbebe May 25, 2025 Author

Uh oh!

winterhazel May 25, 2025 Collaborator

TadiosAbebe
May 24, 2025

Replies: 1 comment 3 replies

winterhazel
May 24, 2025
Collaborator

TadiosAbebe May 24, 2025
Author

TadiosAbebe May 25, 2025
Author

winterhazel May 25, 2025
Collaborator