YARN-11813. Fix the fallback ordering between cgroup v2 and v1. #7631

brumi1024 · 2025-04-17T15:11:46Z

Description of PR

The fallback ordering between cgroup v2 and v1 handlers are now changed, so that now if cgroup v2 is enabled it'll be used first, and if the controller is not mounted in v2 YARN will try to search for it in a v1 hierarchy. This needed some adjustments in the init procedure of the v2 handlers when preconfigured mount paths are used, but the end result should be similar.

How was this patch tested?

Deployed to a cluster, and tested cgroup v1/v2 functionality, by specifically recreating the error scenario described in the ticket.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

...he/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2HandlerImpl.java

K0K0V0K · 2025-04-17T15:21:24Z

Thanks @brumi1024 !

For finding and fixing this case. LGTM!

p-szucs

@brumi1024 Thanks for investigating and fixing this issue! LGTM

hadoop-yetus · 2025-04-17T17:40:50Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 32s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 57s		trunk passed
+1 💚	compile	1m 27s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 24s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	javadoc	0m 47s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 26s		trunk passed
+1 💚	shadedclient	34m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	1m 22s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 22s		the patch passed
+1 💚	compile	1m 18s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 18s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 28s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 33s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 24s		the patch passed
+1 💚	shadedclient	33m 45s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	27m 31s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		147m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/1/artifact/out/Dockerfile
GITHUB PR	#7631
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 33b07835cdc2 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `a5370e8`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/1/testReport/
Max. process+thread count	545 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-04-22T17:49:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 37s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	35m 52s		trunk passed
+1 💚	compile	1m 26s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 20s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 40s		trunk passed
+1 💚	mvnsite	0m 45s		trunk passed
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 40s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 25s		trunk passed
+1 💚	shadedclient	34m 42s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 33s		the patch passed
+1 💚	compile	1m 16s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 16s		the patch passed
+1 💚	compile	1m 13s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 27s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1)
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	33m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	27m 36s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		161m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/2/artifact/out/Dockerfile
GITHUB PR	#7631
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e5ba2db6514a 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ddd9f01`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/2/testReport/
Max. process+thread count	554 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2025-04-24T17:14:41Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 35s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	38m 20s		trunk passed
+1 💚	compile	1m 56s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	1m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 41s		trunk passed
+1 💚	mvnsite	0m 54s		trunk passed
+1 💚	javadoc	0m 53s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 44s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 50s		trunk passed
+1 💚	shadedclient	36m 20s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 34s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	1m 14s		the patch passed
+1 💚	compile	1m 14s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	1m 13s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 26s		the patch passed
+1 💚	mvnsite	0m 35s		the patch passed
+1 💚	javadoc	0m 35s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 32s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 23s		the patch passed
+1 💚	shadedclient	34m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	27m 37s		hadoop-yarn-server-nodemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		153m 35s

Subsystem	Report/Notes
Docker	ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/3/artifact/out/Dockerfile
GITHUB PR	#7631
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5c84b59d3ad8 5.15.0-136-generic #147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `ceb7be7`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/3/testReport/
Max. process+thread count	553 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7631/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

brumi1024 · 2025-04-25T11:55:31Z

Thanks @K0K0V0K and @p-szucs for the review, merging to trunk.

…he#7631)

YARN-11813. Fix the fallback ordering between cgroup v2 and v1.

a5370e8

github-actions bot added YARN trunk labels Apr 17, 2025

K0K0V0K reviewed Apr 17, 2025

View reviewed changes

...he/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsV2HandlerImpl.java Outdated Show resolved Hide resolved

K0K0V0K approved these changes Apr 17, 2025

View reviewed changes

p-szucs approved these changes Apr 17, 2025

View reviewed changes

Update based on comments.

ddd9f01

Fix linting issues.

ceb7be7

brumi1024 merged commit 189c8b6 into apache:trunk Apr 25, 2025
1 of 3 checks passed

brumi1024 deleted the YARN-11813 branch April 25, 2025 11:55

zhtttylz pushed a commit to zhtttylz/hadoop that referenced this pull request Apr 27, 2025

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. (apac…

b301cc1

…he#7631)

zhtttylz pushed a commit to zhtttylz/hadoop that referenced this pull request Apr 29, 2025

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. (apac…

157ea5f

…he#7631)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. #7631

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. #7631

Uh oh!

brumi1024 commented Apr 17, 2025 •

edited

Loading

Uh oh!

Uh oh!

K0K0V0K commented Apr 17, 2025

Uh oh!

p-szucs left a comment

Uh oh!

hadoop-yetus commented Apr 17, 2025

Uh oh!

hadoop-yetus commented Apr 22, 2025

Uh oh!

hadoop-yetus commented Apr 24, 2025

Uh oh!

brumi1024 commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. #7631

YARN-11813. Fix the fallback ordering between cgroup v2 and v1. #7631

Uh oh!

Conversation

brumi1024 commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

How was this patch tested?

For code changes:

Uh oh!

Uh oh!

K0K0V0K commented Apr 17, 2025

Uh oh!

p-szucs left a comment

Choose a reason for hiding this comment

Uh oh!

hadoop-yetus commented Apr 17, 2025

Uh oh!

hadoop-yetus commented Apr 22, 2025

Uh oh!

hadoop-yetus commented Apr 24, 2025

Uh oh!

brumi1024 commented Apr 25, 2025

Uh oh!

Uh oh!

Uh oh!

brumi1024 commented Apr 17, 2025 •

edited

Loading