Add chatgpt powered explanations to Flamegraphs

[Rperry2174]

For the Grafana hackathon we are exploring weather or not ai-based flamegraph explanations would make Pyroscope easier to use. Let us know your thoughts after you've tried it out!

[niufei8888]

This is a very useful feature! Can you help explain more about how this is integrated with ChatGPT?

[bryanhuhta]

I partially answered this in a more recent question #2797 (comment), but I'll expand a bit here as well. Be sure to read the docs too!

We not only can request DOT format flame graphs and funnel them into a ChatGPT (or really any LLM prompt), but we can also do this with the source code itself if you have the GitHub integration configured. This will allow us to map a profile to your source code and annotate the cost of each line. With this feature enabled, the AI can actually provide you recommendations on source code changes in response to the profiling information it's aware of.

[AlJohri]

I see this blogpost: https://pyroscope.io/blog/ai-powered-flamegraph-interpreter/

But I don't see any "Explain Flamegraph" or "ai explainer" button on https://flamegraph.com/

[bryanhuhta]

Explain Flame Graph is only available through to Profiles Drilldown (formerly Explore Profiles) app plugin: https://grafana.com/docs/grafana/latest/explore/simplified-exploration/profiles/.

Repo: https://github.com/grafana/explore-profiles

[EdwaRen]

Hey @Rperry2174, curious how you you fed a flamegraph as an input into an LLM. Did you tell ChatGPT to analyze a screenshot or a massive CSV of every row in the graph?

[bryanhuhta]

Our AI feature is powered by converting the flame graph in question to DOT format. Pyroscope's /render endpoint can optionally serve the flame graph in DOT format (code). This produces a representation of the flame graph similar to what the pprof tool does with Go and graphviz. Here's an example rendered output of a DOT file.

From there, we feed the raw DOT file into a prompt. There are actually several different ones that we will use in different scenarios.

If you're using Grafana Cloud, we will leverage the LLM plugin in your Grafana Cloud instance to serve the prompt. If you're running OSS, then you'll need to bring your own LLM key.

[EdwaRen]

@bryanhuhta how effective is this tool in hindsight now that it's been available for over a year? From my experience LLMs are not good at data-analysis but that was almost a year ago and I was using the CSV format for flame graphs.

Full transparency, I'm not a user of Grafana but I work often with flame graphs.

This is really cool work, I haven't found anyone doing anything like this except here.

[bryanhuhta]

It suffers from a pretty common problem that a lot of general purpose models have, which is that the LLM wants to have an answer even if there is none. So for mature projects, where the hot path has had potentially years to be optimized and tuned, the LLM struggles to provide meaningful suggestions. Nonetheless, it will still try provide some suggestion which is often of dubious quality.

For less optimized code, it actually does alright. I have no numbers to back this up and we haven't done a formal evaluation. However, I have personal experience seeing it catch things like suggesting to lift a static check outside a loop or observing a function is calling malloc a lot and recommending pretty valid remediations (e.g. to use pooled memory).

In general, it's reasonably effective at catching so-called "rookie mistakes." Trying to do real, heavy-duty optimization is definitely a gap. That said, one could argue that performing intense optimization will always require significant domain knowledge and that domain knowledge is hard to capture in an LLM prompt.