Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[native] Add LinuxMemoryChecker check/warning to ensure system-mem-limit-gb is reasonably set #24149

Merged
merged 1 commit into from
Feb 27, 2025
Merged

[native] Add LinuxMemoryChecker check/warning to ensure system-mem-limit-gb is reasonably set #24149

merged 1 commit into from
Feb 27, 2025
+418 −132

Conversation

minhancao
Copy link
Contributor

@minhancao minhancao commented Nov 26, 2024 •
edited
Loading

Description

Add additional checks and warnings to ensure
system-memory-gb <= system-mem-limit-gb < available machine memory of deployment.

For cgroup v1:
Set available machine memory of deployment to be the smaller number
between /proc/meminfo and memory.limit_in_bytes.

For cgroup v2:
Set available machine memory of deployment to be the smaller number
between /proc/meminfo and memory.max.
If memory.max contains "max" string, then look at
/proc/meminfo for the MemTotal, otherwise use the
value in memory.max.

Motivation and Context

Impact

Test Plan

Contributor checklist

  • Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
  • PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
  • Documented new properties (with its default value), SQL syntax, functions, or other functionality.
  • If release notes are required, they follow the release notes guidelines.
  • Adequate tests were added if applicable.
  • CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

Prestissimo (Native Execution)
* Replace default values for configuration properties ``system-memory-gb`` to 57 
  and ``system-mem-limit-gb`` to 60.

Documentation
* Replace default values for configuration properties ``system-memory-gb`` to 57 
  and ``system-mem-limit-gb`` to 60.

Sorry, something went wrong.

@minhancao minhancao self-assigned this Nov 26, 2024
@prestodb-ci prestodb-ci added the from:IBM PR from IBM label Nov 26, 2024
@prestodb-ci prestodb-ci requested review from a team, psnv03 and pramodsatya and removed request for a team November 26, 2024 02:04
@minhancao minhancao marked this pull request as ready for review November 26, 2024 02:07
@minhancao minhancao requested a review from a team as a code owner November 26, 2024 02:07
@minhancao minhancao changed the title [native] Add LinuxMemoryChecker warnings to ensure system-memory-gb < system-mem-limit-gb < actual total memory capacity [native] Add LinuxMemoryChecker warnings to ensure system-mem-limit-gb is reasonably set Nov 26, 2024
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 4478ae1 to 15f55bb Compare November 26, 2024 02:29
@minhancao minhancao changed the title [native] Add LinuxMemoryChecker warnings to ensure system-mem-limit-gb is reasonably set [native] Add LinuxMemoryChecker check/warning to ensure system-mem-limit-gb is reasonably set Nov 26, 2024
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 15f55bb to 7646600 Compare November 26, 2024 06:06
czentgr
czentgr reviewed Nov 27, 2024
View reviewed changes
Copy link
Contributor

@czentgr czentgr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test with fake files again just like we did with the original tests for this class?
That way we can try the "max" value for cgv2, and gigantic values and reasonable values. Basically testing the various situations we saw when investigating this.

@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch 2 times, most recently from 8da401b to 4ae2cee Compare December 3, 2024 20:08
pramodsatya
pramodsatya reviewed Dec 10, 2024
View reviewed changes
Copy link
Contributor

@pramodsatya pramodsatya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @minhancao, could you please squash the commits?

@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 85b3b9d to dab2335 Compare December 13, 2024 00:00
majetideepak
majetideepak previously requested changes Jan 2, 2025
View reviewed changes
Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is ideal if we can avoid checking in data files for testing.
We only need a few fields from the file for testing.
Can we write these required fields to a temporary file as part of the testing?

@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch 2 times, most recently from 71c9fa9 to 89a50a8 Compare January 16, 2025 23:53
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 89a50a8 to 163880b Compare January 22, 2025 18:16
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 163880b to 29dc3b5 Compare January 28, 2025 00:40
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch 2 times, most recently from 9a7cbf7 to 80ca9b5 Compare February 6, 2025 22:25
@minhancao
Copy link
Contributor Author

@czentgr @majetideepak @pramodsatya
I have addressed all the PR comments, please review this PR when you can, thank you!

czentgr
czentgr previously approved these changes Feb 11, 2025
View reviewed changes
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 822c4e1 to 4d982e6 Compare February 13, 2025 20:21
@minhancao
Copy link
Contributor Author

@jaystarshot Hi Jay! Can you help review this PR when you get the chance please?

@jaystarshot
Copy link
Member

Please add some release notes since default values have changed

@minhancao minhancao dismissed stale reviews from majetideepak and czentgr via 79d104c February 18, 2025 18:45
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 4d982e6 to 79d104c Compare February 18, 2025 18:45
jaystarshot
jaystarshot previously approved these changes Feb 18, 2025
View reviewed changes
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 79d104c to f3920ad Compare February 18, 2025 22:13
majetideepak
majetideepak previously approved these changes Feb 18, 2025
View reviewed changes
@majetideepak
Copy link
Collaborator

@minhancao CI is failing. Please verify before merging.

@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from f3920ad to 7c8e159 Compare February 19, 2025 02:42
@minhancao minhancao dismissed stale reviews from majetideepak and jaystarshot via fa8f2dc February 19, 2025 23:25
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch 4 times, most recently from 4df91c3 to e4c2d31 Compare February 25, 2025 00:17
@minhancao
Copy link
Contributor Author

@czentgr @majetideepak @jaystarshot
I have fixed the CI failure. Please review this PR one more time, thank you!

@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from e4c2d31 to 6271440 Compare February 26, 2025 20:31
@minhancao
[native] Add LinuxMemoryChecker check/warning to ensure system-mem-li…

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode
Loading
Loading status checks…
8ea4c76 
…mit-gb is reasonably set

Add additional checks and warnings to ensure
system-memory-gb <= system-mem-limit-gb < available machine memory of deployment.

For cgroup v1:
Set available machine memory of deployment to be the smaller number
between /proc/meminfo and memory.limit_in_bytes.

For cgroup v2:
Set available machine memory of deployment to be the smaller number
between /proc/meminfo and memory.max.
If memory.max contains "max" string, then look at
/proc/meminfo for the MemTotal, otherwise use the
value in memory.max.
@minhancao minhancao force-pushed the linuxmemorychecker_mem_limit_check branch from 6271440 to 8ea4c76 Compare February 26, 2025 21:12
@majetideepak
Copy link
Collaborator

@jaystarshot can you please approve this PR again? CI is now passing.

@czentgr czentgr merged commit 40fe92e into prestodb:master Feb 27, 2025
61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pramodsatya pramodsatya

@majetideepak majetideepak

@jaystarshot jaystarshot

@czentgr czentgr

@steveburnett steveburnett

@psnv03 psnv03

@elharo elharo

@presto-oss presto-oss

Assignees

@minhancao minhancao

Labels
from:IBM PR from IBM
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@minhancao @jaystarshot @majetideepak @steveburnett @czentgr @pramodsatya @prestodb-ci