HDFS-17772. Fix JournaledEditsCache int overflow while the maximum capacity to be Integer MAX_VALUE. #7617
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…g the maximum capacity to always be Integer MAX_VALUE
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Hi Sirs @Hexiaoqiao @ayushtkn @goiri @ZanderXu @simbadzina @slfan1989 Could you please help me review this pr when you have free time ? Thanks a lot~ |
Hexiaoqiao
approved these changes
Apr 18, 2025
|
Thank you very much for your review. @Hexiaoqiao Thank you! |
Hexiaoqiao
changed the title
|
Committed to trunk. Thanks @gyz-web for your contribution. |
|
Thanks a lot for merging to trunk, @Hexiaoqiao thank you very much. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA: HDFS-17772.
Description of PR
When we use the RBF SBN READ in the production environment, we found the following issue.
HDFS-16550 provides the parameter
dfs.journalnode.edit-cache-size.fractionto control cache size based on journalnode memory ratio, but there is an issue of int overflow:When using the
dfs.journalnode.edit-cache-size.fractionparameter to control cache capacity, during the initialization of thecapacityinorg.apache.hadoop.hdfs.qjournal.server.JournaledEditsCache#JournaledEditsCache, a long-to-int overflow issue occurs. For instance, when the heap memory is configured as 32GB (whereRuntime.getRuntime().maxMemory()returns 30,542,397,440 bytes), the overflow results in thecapacitybeing truncated toInteger.MAX_VALUE(2,147,483,647).This renders the parameter setting ineffective, as the intended proportional cache capacity cannot be achieved. To resolve this, the
capacityshould be declared as alongtype, and thetotalSizevariable should also be converted to alongtype to prevent overflow in scenarios wherecapacityexceeds 2,147,483,647, ensuring both variables can accurately represent large values without integer limitations.The error situation is as follows:
The dfs.Journalnode.edit-cache-size.fraction parameter uses the default value of 0.5f. I configured the heap memory size of Journalnode to 30,542,397,440 bytes, and expected it to be 15,271,198,720 bytes, but the capacity size is always Integer.MAX_VALUE=2,147,483,647 bytes
2025-04-15 14:14:03,970 INFO server.Journal (JournaledEditsCache.java:<init>(144)) - Enabling the journaled edits cache with a capacity of bytes: 2147483647The repaired result is as follows，meet expectation:
2025-04-15 16:04:44,840 INFO server.Journal (JournaledEditsCache.java:<init>(144)) - Enabling the journaled edits cache with a capacity of bytes: 15271198720How was this patch tested?
Since Runtime.getRuntime().maxMemory() cannot be adjusted in unit tests, it is not easy to write unit tests, but code changes are no problem for existing unit tests.