[ML] Report the "actual" memory usage of the autodetect process #2846

edsavage · 2025-04-04T02:41:11Z

Determine the actual memory usage of the autodetect process as reported by the OS, e.g. on Linux this would be the value of the maximum resident set size returned by a call to getrusage.

Add this value to the model size stats record returned to the ES Java process so it can be included in the job counts tab for anomaly detection jobs.

Relates elastic/elasticsearch#126256

Determine the actual memory usgae of the autodetect process as reported by the OS, e.g. on Linux this mould be the value of the maximum resident set size returned by a call to `getrusage`. Add this value to the model size stats record returned to the ES Java process so it can be included in the `job counts` tab for anomaly detection jobs.

valeriy42

Hi Ed. I did the first pass.

We should discuss the naming of the new field. While "actual" conveys the intention of the value, it is confusing to the user.

Also, does maximum resident set size actually correspond to the actual current memory usage or is it the historical peak process memory usage?

valeriy42 · 2025-04-08T14:04:37Z

include/model/CResourceMonitor.h

@@ -180,6 +181,8 @@ class MODEL_EXPORT CResourceMonitor {
    //! Returns the sum of used memory plus any extra memory
    std::size_t totalMemory() const;

+    std::size_t actualMemoryUsage() const;


I think we can come up with something better than actualMemoryUsage. Maybe: systemMemoryUsage?

bin/autodetect/Main.cc

edsavage · 2025-04-08T22:30:32Z

Also, does maximum resident set size actually correspond to the actual current memory usage or is it the historical peak process memory usage?

The resident set size (RSS) represents the process's current RAM usage (so not counting pages that have been swapped out etc.), and the max RSS is the high water mark of that value. I think that reporting both would be useful for our purposes.

* ActualMemory -> SystemMemory * Report current resident set size as well as max

bin/autodetect/Main.cc

include/model/CResourceMonitor.h

valeriy42 · 2025-04-10T12:01:24Z

include/model/ModelTypes.h

+    E_AssignmentBasisSystemMemoryBytes = 4, //!< Use the current system memory size
+    E_AssignmentBasisMaxSystemMemoryBytes = 5 //!< Use the highest ever system memory size


I don't think we need this assignment basis reasons (at least so far)

valeriy42 · 2025-04-10T12:02:26Z

bin/autodetect/Main.cc

+        ml::counter_t::E_TSADResidentSetSize,
+        ml::counter_t::E_TSADMaxResidentSetSize};


Should we call these counters SystemMemoryUsage for consistency?

valeriy42 · 2025-04-10T12:05:21Z

lib/model/ModelTypes.cc

+    case E_AssignmentBasisSystemMemoryBytes:
+        return "system_memory_bytes";
+    case E_AssignmentBasisMaxSystemMemoryBytes:
+        return "max_system_memory_bytes";


Are those necessary?

Co-authored-by: Valeriy Khakhutskyy <[email protected]>

…mem_usage # Conflicts: # bin/autodetect/Main.cc # include/model/CResourceMonitor.h

elastic-sonarqube · 2025-04-11T03:31:21Z

Quality Gate failed

Failed conditions
274 New issues
23.0% Duplication on New Code (required ≤ 3%)
164 New Major Issues (required ≤ 0)
10 New Critical Issues (required ≤ 0)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

prodsecmachine · 2025-05-25T22:01:30Z

🎉 Snyk checks have passed. No issues have been found so far.

✅ security/snyk check is complete. No issues have been found. (View Details)

✅ license/snyk check is complete. No issues have been found. (View Details)

* Address failing unit tests * More accurate, meaningful description of new program counters

valeriy42

Looks good. I think the last piece that is missing now is that we use the system memory usage when in CResourceMemory when calculating if allocations are allowed and that we report it back to Java as "model memory usage" and "peak model memory usage" instead of the estimated values on Linux.

… set size) for the "model memory usage" and "peak model memory usage" fields reported to Java.

edsavage · 2025-06-05T21:52:26Z

buildkite run_qa_tests

valeriy42 · 2025-06-06T13:51:16Z

lib/model/CResourceMonitor.cc

I think, this change is missing to update CResrouceMonitor::totalMemory(). Please, correct me if I'm wrong, but the way I understand this code, totalMemory() on Linux should now simply return systemMemoryUsage().

valeriy42 · 2025-06-06T13:54:18Z

lib/model/CResourceMonitor.cc

    res.s_Usage = this->totalMemory();
-    res.s_AdjustedUsage = this->adjustedUsage(res.s_Usage);
+    res.s_AdjustedUsage = systemMemoryUsage(this->adjustedUsage(res.s_Usage));


I don't think we need this complexity here. We can keep reporting adjusted usage the way we did before. But we need to update the value for s_Usage. If you change the logic in totalMemory(), you don't need to change anything here.

Thanks for the comments Valeriy.

What I was thinking was since s_AdjustedUsage and s_AdjustedPeakUsage are used as the values for the fields MODEL_BYTES and PEAK_MODEL_BYTES respectively -

ml-cpp/lib/api/CModelSizeStatsJsonWriter.cc

Lines 57 to 61 in c56874a

writer.onKey(MODEL_BYTES);

writer.onUint64(results.s_AdjustedUsage);

writer.onKey(PEAK_MODEL_BYTES);

writer.onUint64(results.s_AdjustedPeakUsage);

that on Linux these should simply both be set to systemMemoryUsage() and left as-is elsewhere. In doing so I was trying to avoid "adjusting" the system memory usage as it should not be necessary to do so.

That said, I'll adjust the code as you suggest and see how the results look.

On Linux, use systemMemoryUsage for the value of totalMem. Do not "adjust" this value as is done for the estimated usage, as it is unnecessary.

elastic-sonarqube · 2025-06-09T04:36:34Z

Quality Gate failed

Failed conditions
1 New issue
1 New Critical Issues (required ≤ 0)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

edsavage added >enhancement :ml v9.1.0 labels Apr 4, 2025

edsavage added 4 commits April 4, 2025 15:53

Update changelog

cb957ca

Formatting

8f73f02

Appease SonarQube

d3a39ae

Tweak unit test for platform portability

e2d1bf5

edsavage marked this pull request as ready for review April 8, 2025 02:03

edsavage mentioned this pull request Apr 8, 2025

[ML] Accommodate changes to calculation of "model bytes" and "peak mode bytes" values.. elastic/elasticsearch#126256

Open

edsavage requested a review from valeriy42 April 8, 2025 02:05

valeriy42 reviewed Apr 8, 2025

View reviewed changes

Attend to review comments

1a9a99a

* ActualMemory -> SystemMemory * Report current resident set size as well as max

valeriy42 reviewed Apr 10, 2025

View reviewed changes

edsavage and others added 6 commits April 11, 2025 11:08

Update bin/autodetect/Main.cc

5ae22cb

Co-authored-by: Valeriy Khakhutskyy <[email protected]>

Update include/model/CResourceMonitor.h

fe6f1fa

Co-authored-by: Valeriy Khakhutskyy <[email protected]>

Update include/model/CResourceMonitor.h

582430e

Co-authored-by: Valeriy Khakhutskyy <[email protected]>

Attend to review comments

9476ede

Merge remote-tracking branch 'origin/ad_real_mem_usage' into ad_real_…

e29d8f7

…mem_usage # Conflicts: # bin/autodetect/Main.cc # include/model/CResourceMonitor.h

Formatting

475fef1

Merge branch 'main' of github.com:elastic/ml-cpp into ad_real_mem_usage

fc8888c

edsavage added 2 commits May 27, 2025 15:50

Attend to failing unit tests (hopefully)

6945d3f

* Address failing unit tests * More accurate, meaningful description of new program counters

Formatting.. grr

3b69b72

valeriy42 reviewed Jun 2, 2025

View reviewed changes

edsavage added 3 commits June 4, 2025 16:24

On Linux only, use the value of the system memory usage (max resident…

1bafa8c

… set size) for the "model memory usage" and "peak model memory usage" fields reported to Java.

Fix copyright headers

efc311b

Nits in test code

6be6395

edsavage added the ci:run-qa-tests Run a subset of the QA tests label Jun 5, 2025

valeriy42 reviewed Jun 6, 2025

View reviewed changes

edsavage added 2 commits June 9, 2025 16:06

Attend to code review comments

134a494

On Linux, use systemMemoryUsage for the value of totalMem. Do not "adjust" this value as is done for the estimated usage, as it is unnecessary.

Formatting

ed426ac

		E_AssignmentBasisSystemMemoryBytes = 4, //!< Use the current system memory size
		E_AssignmentBasisMaxSystemMemoryBytes = 5 //!< Use the highest ever system memory size

		ml::counter_t::E_TSADResidentSetSize,
		ml::counter_t::E_TSADMaxResidentSetSize};

	writer.onKey(MODEL_BYTES);
	writer.onUint64(results.s_AdjustedUsage);

	writer.onKey(PEAK_MODEL_BYTES);
	writer.onUint64(results.s_AdjustedPeakUsage);

[ML] Report the "actual" memory usage of the autodetect process #2846

Are you sure you want to change the base?

[ML] Report the "actual" memory usage of the autodetect process #2846

Conversation

edsavage commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

edsavage commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

valeriy42 Apr 10, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Apr 11, 2025

Quality Gate failed

Uh oh!

prodsecmachine commented May 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 Snyk checks have passed. No issues have been found so far.

Uh oh!

valeriy42 left a comment

Choose a reason for hiding this comment

Uh oh!

edsavage commented Jun 5, 2025

Uh oh!

valeriy42 Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

valeriy42 Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

edsavage Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

elastic-sonarqube bot commented Jun 9, 2025

Quality Gate failed

Uh oh!

Uh oh!

edsavage commented Apr 4, 2025 •

edited

Loading

prodsecmachine commented May 25, 2025 •

edited

Loading