Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cgroup "anyController" is null in New Docker Engine #1129

Closed
flexdeviser opened this issue Feb 6, 2025 · 5 comments
Closed

Cgroup "anyController" is null in New Docker Engine #1129

flexdeviser opened this issue Feb 6, 2025 · 5 comments

Comments

@flexdeviser
Copy link

Agent: jmx_prometheus_javaagent-1.1.0.jar

There is no problem with the old version of Docker

Docker Engine v27.4.0. works ok.

but after I upgrade to 27.5.1, it start throw exceptions. looks like something related to Cgroup changes in Docker.
Docker Engine v27.5.1. throw exception

2025-02-07 10:17:36 Failed to start Prometheus JMX Exporter ... 2025-02-07 10:17:36 2025-02-07 10:17:36 java.lang.NullPointerException: Cannot invoke "jdk.internal.platform.CgroupInfo.getMountPoint()" because "anyController" is null 2025-02-07 10:17:36 at java.base/jdk.internal.platform.cgroupv2.CgroupV2Subsystem.getInstance(CgroupV2Subsystem.java:80) 2025-02-07 10:17:36 at java.base/jdk.internal.platform.CgroupSubsystemFactory.create(CgroupSubsystemFactory.java:114) 2025-02-07 10:17:36 at java.base/jdk.internal.platform.CgroupMetrics.getInstance(CgroupMetrics.java:177) 2025-02-07 10:17:36 at java.base/jdk.internal.platform.SystemMetrics.instance(SystemMetrics.java:29) 2025-02-07 10:17:36 at java.base/jdk.internal.platform.Metrics.systemMetrics(Metrics.java:58) 2025-02-07 10:17:36 at java.base/jdk.internal.platform.Container.metrics(Container.java:43) 2025-02-07 10:17:36 at jdk.management/com.sun.management.internal.OperatingSystemImpl.<init>(OperatingSystemImpl.java:182) 2025-02-07 10:17:36 at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl.getOperatingSystemMXBean(PlatformMBeanProviderImpl.java:280) 2025-02-07 10:17:36 at jdk.management/com.sun.management.internal.PlatformMBeanProviderImpl$3.nameToMBeanMap(PlatformMBeanProviderImpl.java:199) 2025-02-07 10:17:36 at java.management/sun.management.spi.PlatformMBeanProvider$PlatformComponent.getMBeans(PlatformMBeanProvider.java:195) 2025-02-07 10:17:36 at java.management/java.lang.management.ManagementFactory.getPlatformMXBean(ManagementFactory.java:687) 2025-02-07 10:17:36 at java.management/java.lang.management.ManagementFactory.getOperatingSystemMXBean(ManagementFactory.java:389) 2025-02-07 10:17:36 at io.prometheus.metrics.instrumentation.jvm.ProcessMetrics$Builder.register(ProcessMetrics.java:303) 2025-02-07 10:17:36 at io.prometheus.metrics.instrumentation.jvm.JvmMetrics$Builder.register(JvmMetrics.java:67) 2025-02-07 10:17:36 at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:73) 2025-02-07 10:17:36 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2025-02-07 10:17:36 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) 2025-02-07 10:17:36 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 2025-02-07 10:17:36 at java.base/java.lang.reflect.Method.invoke(Method.java:568) 2025-02-07 10:17:36 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:491) 2025-02-07 10:17:36 at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:503) 2025-02-07 10:17:36 2025-02-07 10:17:36 Prometheus JMX Exporter exiting

@flexdeviser
Copy link
Author

Someone says, it's the problem of updating the Linux Kernal verison to 6.12.*, I checked it in my docker container I see

Image

@dhoard
Copy link
Collaborator

dhoard commented Feb 7, 2025

Can you publish a container image or provide the Dockerfile use to build the image?

@flexdeviser
Copy link
Author

There is no issue with the image I'm using.

You can see I can run the same image with docker engine 27.4.0 without any problem.

Docker itself isn't tied to any specific version of the linux kernel. The container communicates with whatever kernel the host is running

When I run Docker on OS X, it will start a linux VM called [Moby](https://github.com/moby/moby) , and that VM runs a specific kernel version

Dockerfile.txt

@dhoard
Copy link
Collaborator

dhoard commented Feb 7, 2025

Then underlying issue isn’t obvious/may be a combination of Moby + OS X.

The export uses standard Java code/methods.

The full integration test suite has been tested/passes on an Ubuntu 24.0.1 host + Docker 27.5.1 using 98 Java Docker containers (https://github.com/prometheus/jmx_exporter/blob/main/integration_test_suite/integration_tests/src/test/resources/java-docker-images.txt)

More investigation is needed.

@flexdeviser
Copy link
Author

@dhoard no need to investigate anymore. I found the issue. I upgrade my jdk 17 to latest version and works fine. java-17-amazon-corretto-jdk=1:17.0.14.7-1. I can see the metrics now. Thanks for your help.

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants