feat: profile memory of command #164

not-matthias · 2025-11-24T09:55:30Z

Changes in this PR:

Add MessagePack serialization for markers and heaptrack results
Made the RunnerFifo shared across modes (walltime and memory use it at the moment)
Added the heaptrack CLI + support to enable/disable it via IPC
Added the memory executor (WIP)

TODO:

Create release CI for codspeed-heaptrack
Fix lints
Add upload implementation for memory
~~Parse the backend response~~ skipped for now, we'll add this later
Ensure env forwarding works (cargo codspeed run)
Rename to memtrack

Copilot

Pull request overview

This pull request adds memory profiling capability to the codebase through a new Memory executor that uses eBPF-based tracking via a custom heaptrack crate. The implementation allows tracking memory allocations (malloc, free, calloc, realloc, etc.) in benchmarked programs.

Key Changes:

Introduces a new MemoryExecutor that uses eBPF uprobes to track memory allocations
Adds a heaptrack crate with BPF programs for monitoring malloc/free and related syscalls
Refactors FIFO communication handling into shared code to support multiple executor types
Updates test snapshots to use Git LFS for large binary data

Reviewed changes

Copilot reviewed 52 out of 54 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`src/run/runner/memory/executor.rs`	Implements the new Memory executor using heaptrack IPC for control
`src/run/runner/shared/fifo.rs`	Extracts shared FIFO message handling logic for reuse across executors
`src/run/runner/wall_time/perf/mod.rs`	Refactors to use shared FIFO handling code
`crates/heaptrack/*`	New crate providing eBPF-based memory tracking with BPF programs and IPC
`src/run/uploader/upload.rs`	Adds Memory executor case (currently unimplemented)
`flake.nix`	Adds Nix development environment configuration
`.github/workflows/ci.yml`	Updates CI to install dependencies and run BPF tests separately
Snapshot files	Migrates large test snapshots to Git LFS

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review for a chance to win a $100 gift card. Take the survey.

src/run/uploader/upload.rs

Copilot · 2025-11-25T10:52:41Z

src/run/runner/shared/fifo.rs

+        loop {
+            if self.ctl_fifo.read_exact(&mut buffer).await.is_ok() {
+                break;
+            }
+        }


This infinite loop will never exit if read_exact continuously fails. The loop should either have a timeout, a maximum retry count, or handle/propagate the error. This could lead to the process hanging indefinitely.

GuillaumeLagrange

CI does not build, and the CI change commit scares me a bit, going forward not gonna lie

Just so I know, have you had the protobuf decision challenged ?
Could be good to have a small design document just to make sure the whole team is aligned behind the choice.

flake.nix

src/run/runner/shared/fifo.rs

crates/runner-shared/src/benchmark_results.rs

src/run/mod.rs

crates/runner-shared/Cargo.toml

not-matthias · 2025-12-03T13:33:01Z

@GuillaumeLagrange As discussed, I updated the benchmark result types to use rmp-serde. The file names are now also based on the struct name:

[14:28] not-matthias@~ > eza -alh /tmp/profile.Hrgnlk6YFL.out/results
Permissions Size User         Date Modified Name
.rw-r--r--  658k root          3 Dec 14:28  592283.HeaptrackResult.msgpack
.rw-r--r--  2.4k not-matthias  3 Dec 14:28  MarkerResult.msgpack

GuillaumeLagrange

Really cool! Let me know if you want to discuss naming

crates/runner-shared/src/benchmark_results.rs

GuillaumeLagrange · 2025-12-03T13:46:31Z

crates/runner-shared/src/benchmark_results.rs

+
+#[derive(Debug, Clone, Copy, Serialize, Deserialize)]
+pub struct HeaptrackEvent {
+    pub pid: u32,


use pid_t

I think it's finally time to export a pid_t from runner-shared and use it from here

GuillaumeLagrange · 2025-12-03T14:07:03Z

crates/runner-shared/src/benchmark_results.rs

+use serde::{Deserialize, Serialize};
+
+#[derive(Debug, Clone, Serialize, Deserialize)]
+pub struct MarkerResult {


I get the initial idea, but MarkerResult does feel a bit weird, especially since this struct has both the markers and the uri by timestamp.

How about ExecutionTimestamps or something like this ?

GuillaumeLagrange · 2025-12-03T14:11:02Z

crates/runner-shared/src/benchmark_results.rs

+    Brk { size: u64 },
+}
+
+pub trait BenchmarkResultExt


I think what we are defining here is "A benchmark execution output that we persist to file"

I like Artifact to convey the idea of persisting this to file rather than Result, WDYT ?
And I think Execution rather than Benchmark is more neutral, and technically this is the output of executor.

Since we are persisting these let's try and nail the naming before releasing

GuillaumeLagrange · 2025-12-03T14:38:37Z

crates/heaptrack/build.rs

+        .source("src/bpf/heaptrack.bpf.c")
+        .clang_args([
+            "-I",
+            &vmlinux::include_path_root().join(arch).to_string_lossy(),


If I understand correctly, this builds using bpf headers for the current kernel. Wont that be a huge issue going forward that basically make shipping pre-built heaptrack is going to be impossible ?

GuillaumeLagrange · 2025-12-03T14:40:57Z

crates/heaptrack/src/ebpf/c/heaptrack.bpf.c

+} calloc_size SEC(".maps");
+
+SEC("uprobe")
+int uprobe_calloc(struct pt_regs *ctx) {


same for this file, if I understand correctly it's not generated, we could make this much less repetitive by using proc macros. WDYT ?

This would also ease integration with custom allocator in the future as we could expose the proc macros and clear and documented arguments to each, and the burden of itnegration would be quite small IMO

crates/heaptrack/test.sh

crates/heaptrack/src/lib.rs

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch 3 times, most recently from ca571b9 to 4bdc191 Compare November 24, 2025 11:30

GuillaumeLagrange mentioned this pull request Nov 25, 2025

Feat: add the exec subcommand to run an arbitrary rust binary #165

Open

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch from d168920 to cffcf37 Compare November 25, 2025 10:50

not-matthias requested a review from Copilot November 25, 2025 10:50

Copilot started reviewing on behalf of not-matthias November 25, 2025 10:51 View session

Copilot finished reviewing on behalf of not-matthias November 25, 2025 10:52

Copilot AI reviewed Nov 25, 2025

View reviewed changes

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch 4 times, most recently from 9aaf842 to 28a793b Compare November 26, 2025 13:15

not-matthias requested a review from GuillaumeLagrange November 26, 2025 13:17

GuillaumeLagrange requested changes Nov 27, 2025

View reviewed changes

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch 2 times, most recently from 7f99647 to 3e3e2be Compare December 1, 2025 11:05

feat: add new fifo commands

b614c5e

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch 3 times, most recently from b782293 to ab9e7f8 Compare December 2, 2025 13:08

feat: add shared fifo

ed10354

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch from 0e32eca to 9917752 Compare December 2, 2025 13:48

not-matthias requested a review from GuillaumeLagrange December 2, 2025 13:48

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch 2 times, most recently from b05d317 to 100f423 Compare December 3, 2025 13:30

GuillaumeLagrange requested changes Dec 3, 2025

View reviewed changes

not-matthias added 2 commits December 5, 2025 15:21

feat: add benchmark result types

68a8a68

feat: add heaptrack crate

7e50261

not-matthias added 4 commits December 5, 2025 15:47

feat(heaptrack): allow enabling/disabling via IPC

2285ac6

feat: add memory executor

e0ed9fb

feat: release heaptrack with cargo-dist

0c16b1e

fixup: use workspace deps

e360c02

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch from 100f423 to 863e36a Compare December 5, 2025 14:48

fixup: heaptrack

5f87e90

not-matthias force-pushed the cod-1670-runner-profile-memory-of-command branch from 863e36a to 5f87e90 Compare December 5, 2025 15:00

feat: profile memory of command #164

Are you sure you want to change the base?

feat: profile memory of command #164

Conversation

not-matthias commented Nov 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI Nov 25, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

not-matthias commented Dec 3, 2025

Uh oh!

GuillaumeLagrange left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

not-matthias commented Nov 24, 2025 •

edited

Loading

GuillaumeLagrange left a comment •

edited

Loading