Embedding of Angelscript stack traces into native stack traces

[mikaelkalms-ttkgames]

Hi!

We are using Unreal Engine with Angelscript as an embedded scripting language. This integration was originally done by a game studio called Hazelight, but has since been picked up by Embark Studios and several other game studios. The language offers great ergonomics for development times and iterations, but postmortem debugging is difficult.

The biggest limitation is that the stacktrace that Sentry sees contains no information of what is happening within the Angelscript interpreter. This means that Sentry is unable to take into account what has been happening in Angelscript land when grouping callstacks. It also means that when a person looks at an event, they get an excellent presentation of the native stacktraces + context information, but need to dig a bit to find out anything about the Angelscript side.

We can (and have) improved this a bit by attaching Angelscript-specific context information & printing things to logs ... but it would be really nice to inject Angelscript stackframes into the native callstacks. This would allow Sentry to group events intelligently, taking Angelscript execution into account.

---

A typical stacktrace looks like below. In this example, the code has transitioned C++ => AS => C++ => AS => C++ and then dereferenced a nullptr:

... a bunch of native frames ...
FAngelscriptManager::Execute()
... a bunch of native frames ...
FAngelscriptManager::Execute()
... a bunch of native frames ...
<thread entry point>

In the above example, FAngelscriptManager::Execute() is the entry point for a function call into Angelscript land. There's a bytecode interpreter embedded within the function, with an infinite loop + a switch statement that processes individual instructions of the AngelScript VM. One of those VM instructions is "make a function call into C++", which is how the Angelscript script logic has called back to the C++ side twice in the above example.

---

My first question here is, are there any general strategies to employ when making an interpreted language a "first-class citizen" of events in Sentry?

Secondly, do you have any pointers for this particular case? To narrow things down a bit,

• Angelscript is purely an interpreted language in this scenario; JIT / AOT / transpilation is out of scope
• The Angelscript interpreter has callstack info organized neatly in internal data structures, there are already C++ helper functions to "print Angelscript stacktrace to stdout", which demonstrates that the necessary information exists

---

I have tried retrieving modifying the Sentry Unreal plugin to capture the callstack (tested on Win64) with sentry_value_set_stacktrace(), for crashes, but similar to how FGenericPlatformSentrySubsystem::CaptureMessage() etc does it, identified any occurrences of the return address of FAngelscriptManager::Execute() within that callstack, and inserted Angelscript specific entries according to the Stack Trace Interface format.

This resulted in a datastructure that looked good on paper - at least for the main thread, I didn't attempt to do this for all threads in the program. However, it appears that after the Sentry Unreal plugin uploaded the results to the Sentry backend (again, using Win64, so using crashpad as an intermediary step on the client side, and mechanism = minidump on the backend side), the custom stacktrace I supplied was ignored by the backend.

Perhaps mechanism = minidump means that client-supplied stacktraces will get ignored?

Perhaps I messed up the format?

Perhaps clientside-captured stacktraces are a fool's errand (for example, once one starts to ship executables without complete debug info to customers)?

Any ideas or input would be appreciated!