feat(AI): Microsoft.Extensions.AI Agent Monitoring
#4657
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4657 +/- ##
==========================================
+ Coverage 73.66% 73.79% +0.12%
==========================================
Files 476 485 +9
Lines 17442 17686 +244
Branches 3453 3497 +44
==========================================
+ Hits 12849 13051 +202
- Misses 3744 3774 +30
- Partials 849 861 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
@sentry review |
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
|
@sentry review |
|
@BugBot review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
|
@BugBot review |
|
@sentry review |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
|
@BugBot review |
|
@sentry review |
|
@BugBot review |
|
@sentry review |
|
@BugBot review |
| [Experimental(DiagnosticId.ExperimentalFeature)] | ||
| public static ChatOptions AddSentryToolInstrumentation(this ChatOptions options) | ||
| { | ||
| if (options.Tools is { Count: 0 }) |
There was a problem hiding this comment.
| if (options.Tools is { Count: 0 }) | |
| if (options.Tools is not { Count: > 0 }) |
The above basically reads:
If
options.Toolsis not a non-null object with aCountgreater than zero
The logic you have currently fails to match null values... it only matches a non-null object with a count of zero. That would technically be OK since the loop comparison below would compare i to null and the whole thing would evaluate as false... but:
- We can early exit here already (and avoid that comparison)
- It's not as obvious/explicit that will happen... anyone reading the code needs to do a bit of a double take to work that out
| /// </summary> | ||
| private static readonly ActivityListener FICCListener = new() | ||
| { | ||
| ShouldListenTo = source => source.Name.StartsWith(SentryAIConstants.SentryActivitySourceName), |
There was a problem hiding this comment.
I was expecting the instrumentation for the spans that ME AI adds to be done using this source name:
https://github.com/dotnet/extensions/blob/e7423719f55472f16e61150413fd107ea504c369/src/Libraries/Microsoft.Extensions.AI/OpenTelemetryConsts.cs#L11
... but even that we might not be able to rely on, since it appears people can override the source name for some reason:
https://github.com/dotnet/extensions/blob/f8f779a6ea004bb1f26649719ca77d63a9d9417c/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClientBuilderExtensions.cs#L29
Is that why you've overriden SentryChatClient.GetService here:
sentry-dotnet/src/Sentry.Extensions.AI/SentryChatClient.cs
Lines 104 to 107 in d13640c
It's an interesting solution... I'm wondering if there might be side effects, like if other people's code is also listening for those events and assuming they will have the original ActivitySource name. Given that it can be overriden, maybe they shouldn't be doing that anyway.
@Flash0ver thoughts?
There was a problem hiding this comment.
Ah... it starts to make more sense now I read this:
So yeah, this looks like a nice solution.
| { | ||
| /// <summary> | ||
| /// The list of strings which FunctionInvokingChatClient(FICC) uses to start the tool call <see cref="Activity"/>. | ||
| /// These can be found in FunctionInvokingChatClient.cs in GetResponseAsync or GetStreamingResponseAsync function. |
There was a problem hiding this comment.
I was thinking more a permalink like:
| /// These can be found in FunctionInvokingChatClient.cs in GetResponseAsync or GetStreamingResponseAsync function. | |
| /// See: | |
| /// https://github.com/dotnet/extensions/blob/f8f779a6ea004bb1f26649719ca77d63a9d9417c/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs#L272 | |
| /// |
I couldn't find anything using the names "FunctionInvokingChatClient.GetResponseAsync" or
"FunctionInvokingChatClient"... I think you mentioned those came from older versions of the code? Would be good to add permalinks to those as well in that case.
There was a problem hiding this comment.
Yea, I was mentioning that from the official Microsoft Extensions AI codebase. A link would definitely be better..
There was a problem hiding this comment.
The point of passing IHub in as a dependency is to avoid statics (which are tricky to test).
We should remove the static modifier from this class... can probably ditch the Init method in favour of a regular constructor as well.
If there was some need to only have one instance (strictly), we could implement a singleton pattern here (like this).
I don't think that's necessary though...
There was a problem hiding this comment.
I ended up using a singleton pattern because we only want one Listener registered at all times. Otherwise there may be multiple spans created for a single call to the LLMs.
Fixed in 9b440a3
There was a problem hiding this comment.
Probably easier to describe what I mean in a commit 😉
Take a look at that and see if it makes sense... it's a bit tricky because the only way to add a listener is via ActivitySource.AddActivityListener (which is static), but possible to avoid all the other static references when running tests.
There was a problem hiding this comment.
That looks good, using Lazy is something I haven't seen yet, that's interesting!
I changed the capitalization for AI in SentryAIActivityListener in this commit:
jamescrosswell
left a comment
jamescrosswell
left a comment
There was a problem hiding this comment.
Looking heaps better... I think the main outstanding issue now is this one:
#4657 (comment)
|
Seems like this is close to being finished.. (also bugbot didn't find anything so that's a plus? :) ) Please take a look @jamescrosswell @Flash0ver and let me know if anything's off.. |
jamescrosswell
left a comment
jamescrosswell
left a comment
There was a problem hiding this comment.
I can't see any remaining issues with this (assuming you're OK with the changes I made).
@Flash0ver I think it'd be good to get a second set of eyes on this one as there's quite a lot going on in this PR.
jamescrosswell
left a comment
jamescrosswell
left a comment
There was a problem hiding this comment.
Nice work @alexsohn1126 - not a trivial integration so great to get this over the line!
Resolves #3762
Purpose
This PR will add a new SDK - for
Sentry.Extensions.AI.It will allow users to instrument their LLM usage, including any tool calls and LLM API calls.
Microsoft.Extensions.AI.AbstractionsNuGet package is used in this SDK to use their interfaces and types.Usage
In order to use this SDK, the user will have to have a compatible SDK which can instantiate an
IChatClient. In the samples, I have usedMicrosoft.Extensions.AI.OpenAI. Another popular LLM SDK isAnthropic.SDK.There are a few AI-exclusive options the users can change.
IncludeAIRequestMessages- booleantruetrue, we include the request message in the spans we send to Sentry.IncludeAIResponseContent- booleantruetrue, we include the response message in the spans we send to Sentry.AgentName- stringAgentChatClientrepresents.Once the user has set up and sends AI spans to Sentry, they will be able to see it in
Insights > AI(from the sidebar).Screen.Recording.2025-10-31.at.5.39.14.PM.mov
There are useful information about LLM usage in the webpage. Such as how many tokens the user has sent/received, what tool calls were made, how long they tool, etc.