Conversation
…shboard wedge
The `adk_capability` value list on Aria was too coarse: a single
`evaluations` slice covered create/update/delete/validate, and a single
`publishing` slice covered both `push` (upload local edits to Dataverse)
and `publish` (make changes live). That hid the signal PMs actually
need — which specific action drove a change in usage — and it left
several user-facing skills (onboarding, topic review/test, workflow
test) with no wedge at all.
Split the taxonomy so every distinct user-facing ADK command gets its
own slice on the Capability Usage donut:
evaluations -> evaluation_create / evaluation_update /
evaluation_delete / evaluation_validate
publishing -> push (push.py) + publishing (publish.py, unchanged)
(new wedges) -> onboarding, topic_review, topic_test, workflow_test
Notes:
* push.py's `emit_build_start` / `emit_build_complete` /
`emit_agent_deploy` events are re-stamped from
`adk_capability="publishing"` to `"push"`, and push.py now emits a
best-effort `emit_capability_use("push")` at the end of the flow so
push gets a wedge on the Capability Usage donut (mirroring publish.py).
Aria tiles that filter Build Outcomes / Agent Deploy on
`adk_capability="publishing"` will need updating to include both
values (or split into two tiles).
* `evaluate_evals.py` now emits `evaluation_validate` in-process
(previously emitted `evaluations`).
* SKILL.md files for evaluations/{create,update,delete} switch their
`emit_capability.py` argument to the new granular values.
* SKILL.md files for onboarding, topics/review, topics/test, and
workflows/test gain a best-effort telemetry line following the
troubleshoot/SKILL.md pattern.
* `discover` (owned by PR #238) and `planner` (owned by a separate
feature branch) are intentionally not added here — their PRs will
add their own taxonomy entries.
* `flightcheck` remains in the taxonomy but is not part of the
Capability Usage donut split — it has dedicated FlightCheck tiles.
* Tests updated: `test_wired_capabilities_are_in_canonical_list` gains
the new values; the happy-path emit test uses `evaluation_validate`.
The `test_no_caller_passes_a_noncanonical_capability_to_the_shim`
guard from PR #254 continues to catch drift automatically.
ADO: https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/7830948
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 70ddf006-7f8d-48e7-9afa-3fbae73b3864
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Split the ADK
adk_capabilitytaxonomy so every distinct user-facing command has its own slice on the Capability Usage donut. Umbrella labels (a singleevaluationsorpublishingwedge covering multiple commands) hid the signal PMs need to see which specific action drove a change in usage, and left several user-facing skills with no wedge at all.Changes
evaluations(one wedge for create/update/delete/validate)evaluation_create,evaluation_update,evaluation_delete,evaluation_validatepublishing(one wedge covering bothpush.pyandpublish.py)push(push.py — upload local edits to Dataverse) +publishing(publish.py — make changes live)onboarding,topic_review,topic_test,workflow_testpush.py rename
push.py'semit_build_start/emit_build_complete/emit_agent_deployevents are re-stamped fromadk_capability="publishing"to"push", andpush.pynow emits a best-effortemit_capability_use("push")at the end of the flow so push gets its own wedge on the Capability Usage donut (mirroringpublish.py).Not included in this PR
discover— owned by Add /discover skill: admin-run tenant inventory discovery crawler #238. That PR will adddiscovertoADK_CAPABILITIESand wire it intodiscover_inventory.py.planner— owned by a separate feature branch not yet merged. It will add its own taxonomy entry.flightcheck— stays in the taxonomy but is intentionally not part of the Capability Usage donut split; it already has dedicated FlightCheck dashboard panels.Tests
tests/test_adk_telemetry.py::test_wired_capabilities_are_in_canonical_listupdated for the new granular values.evaluation_validate(wasevaluations).test_no_caller_passes_a_noncanonical_capability_to_the_shimguard from Fix publish.py emitting adk_capability=\"publish\" (non-canonical) #254 continues to catch drift automatically — it grepsscripts/*.pyandSKILL.mdforemit_capability.py <cap>invocations and asserts every value is inADK_CAPABILITIES.pytest tests/test_adk_telemetry.py→ 67 passed.ADO
https://o365exchange.visualstudio.com/O365%20Core/_workitems/edit/7830948