[Azure AI Foundry] Add guardrails dashboard #14551
Conversation
muthu-mps
added
enhancement
🚀 Benchmarks reportTo see the full report comment with |
💚 Build Succeeded
History
cc @muthu-mps |
|
|
Package azure_ai_foundry - 0.5.0 containing this change is available at https://epr.elastic.co/package/azure_ai_foundry/0.5.0/ |
There was a problem hiding this comment.
Couple of comments:
1/ The description of the dashboard mentions "This dashboard primarily focuses on logs generated by the Azure AI Foundry service when connected through the API Management service." However, I can see that the dashboard shows a large number of non-log data such as request rates, error rates, model usage, latency which look like metrics. I think we should change this to "This dashboard focuses on logs and metrics generated by the Azure AI Foundry service when accessed via the API Management service only."
2/ Also, in the description body of the dashboard, remove mentions of "advanced monitoring". We can add an explanation of what Azure API Management service is and that the dashboard includes only data (performance, usage, prompts/responses) from using the API Management service. Something along the lines of "Azure API Management is a hybrid, multicloud management platform for APIs across all environments."
2/ It is not clear why we are calling this dashboard "Advanced monitoring" especially since it shows fairly basic, tablestakes data. Perhaps we should keep the previously suggested name focusing on the fact that the data we visualize is only for usage via the Azure API Management service which is the recommended way to use any Azure service anyway. So perhaps rename the dashboard to "API Management"
There was a problem hiding this comment.
A few things:
1/ Replace description text with:
"The dashboard shows costs related to Azure Cognitive Services, including:
- Total costs
- Daily usage costs
Note: to enable collection of cost data, you need to enable the Azure Billing integration"
2/ replace "Total costs" with "Total cost" in the widget
3/ replace "Daily usage costs" with "Daily usage cost"
4/ replace "daily usage costs - by model" with "daily usage cost - by model"
There was a problem hiding this comment.
A few things:
1/ replace header of description with that says "Azure AI Foundry Guardrails (Advanced Monitoring)" with "Azure AI Foundry Guardrails"
2/ why are we using a donut widget for error rate here but different widgets for error rates in the other dashboards?
3/ can we make the section named "Risk categories - by prompt input" more visible? Maybe bigger letters or something that makes it obvious that this is a new / separate section?
4/ replace "severity distribution by categories" with "severity distribution by category"
5/ make sure that the name capitalization of all your y-axis names of your widget is uniform. The guardrails dashboard has small caps in some cases, and in other cases first letter is capitalized. But then if you look at the other dashboards y-axis has first letter capitalization. We should be using this format.
There was a problem hiding this comment.
| 2/ why are we using a donut widget for error rate here but different widgets for error rates in the other dashboards?
This is because this will showcase which error_code happens most of the time. When it comes to guardrails we can show the content_filter error is happening most of the time.
There was a problem hiding this comment.
1/ replace blocked request by category with 'blocked request - by category"
2/ replace serverity distribution by categories with 'Serverity distribution - by category"
3/ first letter capitalization of y-axis names (see previous comment)
There was a problem hiding this comment.
1/ replace "primary metrics" with "Key metrics"
2/ explain that metrics reflect usage via both API Management service and .... (playground? Azure native APIs? Azure REST API? something else?) Can we work on explaining this as it's not clear what are the various ways in which you can generate such metrics
Proposed commit message
This PR adds support for monitoring the guardrails and controls. This is part of the Advanced monitoring capabilities in Azure for AI Foundry when accessed via API Management Services.
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots