Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add symbolize feature for online symbolization #22

Merged
merged 2 commits into from
Feb 21, 2025

Conversation

erikgrinaker
Copy link
Contributor

This patch adds a symbolize feature which performs online symbolization of profiles using the backtrace crate.

This is often much more convenient than having to manually obtain binaries and symbolize the profiles, and matches the behavior of e.g. the pprof-rs CPU profiler. It will also allow emitting e.g. flamegraph SVGs directly, which I'll submit in a follow-up PR.

Online symbolization uses a fair amount of memory for symbol caches, but this is often a worthwhile tradeoff.

@CLAassistant
Copy link

CLAassistant commented Feb 1, 2025

CLA assistant check
All committers have signed the CLA.

@erikgrinaker erikgrinaker force-pushed the symbolize branch 3 times, most recently from 3e91fa8 to f825061 Compare February 1, 2025 14:29
This patch adds a `symbolize` feature which performs online
symbolization of profiles using the `backtrace` crate.

This is often much more convenient than having to manually obtain
binaries and symbolize the profiles, and matches the behavior of e.g.
the `pprof-rs` CPU profiler. It will also allow emitting e.g. flamegraph
SVGs directly, which I'll submit in a follow-up PR.

Online symbolization uses a fair amount of memory for symbol caches, but
this is often a worthwhile tradeoff.
@erikgrinaker
Copy link
Contributor Author

Hi @umanwizard, just a friendly ping in case y'all missed this.

@umanwizard
Copy link
Collaborator

Thanks and sorry for the delay.

Did you test with stack traces that have inlined functions? Did it work correctly / can you provide an example .pprof ?

@erikgrinaker
Copy link
Contributor Author

No worries.

Did you test with stack traces that have inlined functions?

No, I haven't tested it. I think it should mostly just work as long as there's sufficient debug info (backtrace supports it), but I'll verify. And we should probably set Mapping::has_inline_frames in that case.

I'll have a look over the weekend and get back to you.

@erikgrinaker
Copy link
Contributor Author

Confirming that inlined functions work as expected, using:

#[inline(never)]
fn allocate(size: usize) -> Vec<u8> {
    allocate_inline(size)
}

#[inline(always)]
fn allocate_inline(size: usize) -> Vec<u8> {
    println!("allocating {size}b");
    vec![9; size]
}

Which yields profile.pb.gz:

Screenshot 2025-02-21 at 17 01 59

@umanwizard umanwizard merged commit 2973b2f into polarsignals:main Feb 21, 2025
1 check passed
github-merge-queue bot pushed a commit to neondatabase/neon that referenced this pull request Mar 4, 2025
## Problem

The code to generate symbolized pprof heap profiles and flamegraph SVGs
has been upstreamed to the `jemalloc_pprof` crate:

* polarsignals/rust-jemalloc-pprof#22
* polarsignals/rust-jemalloc-pprof#23

## Summary of changes

Use `jemalloc_pprof` to generate symbolized pprof heap profiles and
flamegraph SVGs.

This reintroduces a bunch of internal jemalloc stack frames that we'd
previously strip, e.g. each stack now always ends with
`prof_backtrace_impl` (where jemalloc takes a stack trace for heap
profiling), but that seems ok.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants