Add memory profiling / logging

[andygrove]

What is the problem the feature request solves?

I would like to add a config that enables memory profiling so that we can monitor JVM and native memory usage throughout the lifetime of a Spark session or job. This data should be written out in a structured file format from which we can generate charts.

In JVM side, we can use:

val memoryMXBean = ManagementFactory.getMemoryMXBean
val heap = memoryMXBean.getHeapMemoryUsage
val nonHeap = memoryMXBean.getNonHeapMemoryUsage

In native side, we can use the procfs crate:

let pid = std::process::id();
let process = Process::new(pid as i32).unwrap();
let statm = process.statm().unwrap();

By logging JVM usage and overall process memory information, we can infer how much native memory is used. We can also log how much memory is reserved in the native memory pools and start to see how that aligns with actual usage.

Describe the potential solution

No response

Additional context

No response

[andygrove]

I plan on working on this

[alamb]

It would be amazing to have memory monitoring of native code in datafusion too -- it is an important feature that is currently hard for downstream crates

[alamb]

Related discussion in DataFusion:

• Add Memory Profiling Functionality datafusion#14510