dotnet-test: make code-testing agent tools portable across Copilot CLI, Claude Code & Gemini CLI + add portability check#856
Open
Evangelink wants to merge 4 commits into
Open
Conversation
…ompatible PR #847 added `tools: ["agent", "skill", "read", "search", "edit", "execute"]` to the code-testing-* agents to enable VS Code / Copilot CLI subagent fan-out. Those lowercase aliases map to real tools in VS Code and the Copilot CLI, but Claude Code matches `tools:` against its own vocabulary (Task, Skill, Read, Glob, Grep, Edit, Write, Bash). None of the aliases matched, so when these agents are loaded into Claude Code via --plugin-dir and selected with `claude --agent`, the agent was granted ZERO tools. A tool-less model asked to generate tests emits a textual <tool_call> block and exits after one turn, producing no file changes. Append the Claude Code tool names to each agent's `tools:` list so the same declaration works across all three runtimes (each honors the names it knows and ignores the foreign ones): - Orchestrators (generator, implementer): add Task, Skill, Read, Glob, Grep, Edit, Write, Bash (Task is the Claude Code equivalent of the `agent` fan-out tool). - Workers (researcher, planner, builder, tester, fixer, linter): add Skill, Read, Glob, Grep, Edit, Write, Bash. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the dotnet-test plugin’s code-testing-* agent frontmatter to make their tools: allow-lists compatible with Claude Code by adding Claude’s tool vocabulary alongside the existing VS Code / Copilot CLI aliases.
Changes:
- Expand
tools:lists to include Claude Code tool names (Task,Skill,Read,Glob,Grep,Edit,Write,Bash) while preserving existing lowercase aliases. - Add
Taskspecifically to orchestrator agents to preserve subagent fan-out behavior in Claude Code.
Show a summary per file
| File | Description |
|---|---|
| plugins/dotnet-test/agents/code-testing-generator.agent.md | Adds Claude Code tool names (including Task) to the orchestrator’s allow-list. |
| plugins/dotnet-test/agents/code-testing-implementer.agent.md | Adds Claude Code tool names (including Task) to the orchestrator’s allow-list. |
| plugins/dotnet-test/agents/code-testing-researcher.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
| plugins/dotnet-test/agents/code-testing-planner.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
| plugins/dotnet-test/agents/code-testing-builder.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
| plugins/dotnet-test/agents/code-testing-tester.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
| plugins/dotnet-test/agents/code-testing-fixer.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
| plugins/dotnet-test/agents/code-testing-linter.agent.md | Adds Claude Code tool names to the worker agent allow-list. |
Copilot's findings
Tip
Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Files reviewed: 8/8 changed files
- Comments generated: 8
…ility check Two related follow-ups to the agent tools fix: 1. Address the skill-check review feedback. The validator's BuiltInTools set was missing three legitimate host tool spellings that are not case-insensitive matches of existing entries, so they were flagged as non-built-in: - "write" — Claude Code file-creation tool (Copilot CLI / VS Code: "create") - "agent" — Copilot CLI / VS Code subagent fan-out tool (Claude Code: "task") - "execute" — Copilot CLI / VS Code run-command tool (Claude Code: "bash") "agent" and "execute" were already flagged before this branch (introduced by the fan-out PR); adding them to BuiltInTools clears the pre-existing warnings. 2. Add a cross-host tool portability check (CheckAgentToolPortability) so an agent that declares a capability for only one host is flagged. Tool names are matched case-sensitively (hosts resolve tools by exact spelling), so an agent that lists e.g. only "edit" (Copilot / VS Code) without "Edit"/"Write" (Claude Code) is reported as working on one host and silently tool-less on the other. Findings are advisory (do not fail CI) and allowlistable via "agent-tool-portability:AGENT:capability". Wired into the agents loop in CheckCommand and covered by unit tests. Also make the one existing single-host agent (optimizing-dotnet-performance) portable by adding its Claude Code tool spellings, so the new check reports a clean tree. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Evangelink
requested review from
a team,
JanKrivanek and
ViktorHofer
as code owners
Evangelink
changed the title
Member
Author
|
/evaluate |
Gemini CLI now supports custom subagents defined the same way as Claude Code and the Copilot CLI (Markdown + YAML frontmatter with a tools: list, bundled in an extension's agents/ dir), but it spells its built-in tools in a third, snake_case vocabulary: read_file, replace, write_file, glob, grep_search, run_shell_command. An agent that lists only the Copilot CLI / VS Code or Claude Code spellings is silently tool-less under Gemini. - Extend the ToolCapability model and CheckAgentToolPortability to N hosts and add the Gemini CLI column. 'invoke subagents' and 'invoke skills' have no Gemini tools-level spelling (Gemini delegates via @agent and loads skills implicitly), so their Gemini list is empty and not enforced. - Add the Gemini built-in tool names to BuiltInTools so they are not flagged as non-built-in. - Add the Gemini spellings to the 8 code-testing agents and to optimizing-dotnet-performance so the tri-host check reports a clean tree. - Update unit tests for the three-host model (32 tests pass); full skill-validator check across all 16 plugins passes with 0 findings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
the
pr-state/ready-for-eval
Evangelink
changed the title
Contributor
Skill Validation Results
[1] (Isolated) Quality dropped but weighted score is +17.4% due to: completion (✗ → ✓), tokens (141891 → 109384), time (79.9s → 57.7s), tool calls (14 → 11) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-author
Contributor
|
👋 @Evangelink — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the |
Addresses review feedback: CheckAgentToolPortability treated a host as having a
capability if *any* tool in that host's list was present, which produced false
negatives for capabilities that listed complementary (non-synonym) tools — e.g.
an agent with Claude Code's Glob but not Grep was considered portable for
'search' even though content search was missing.
Split the two combined capabilities so each host's list holds only interchangeable
spellings (any-of is then correct):
- 'search' -> 'find files' (search / Glob / glob) + 'search file contents'
(search / Grep / grep_search)
- 'modify files' -> 'edit files' (edit / Edit / replace) + 'create files'
(create|edit / Write / write_file)
The Copilot CLI's single broad tool (search, edit) maps to both halves, so its
existing single-tool declarations stay portable while a partial Claude/Gemini
declaration (Glob without Grep, Edit without Write) is now flagged. Added a
regression test for the partial-search case; 33 tests pass and the full
skill-validator check across all 16 plugins still reports 0 findings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Member
Author
|
/evaluate |
Contributor
Skill Validation Results
[1] (Isolated) Quality unchanged but weighted score is -3.8% due to: tokens (111853 → 171698), tool calls (11 → 14) Model: claude-opus-4.6 | Judge: claude-opus-4.6 🔍 Full Results - additional metrics and failure investigation steps
▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
github-actions
Bot
added
waiting-on-review
Contributor
|
✅ Evaluation passed for |
AbhitejJohn
approved these changes
Jul 6, 2026
AbhitejJohn
left a comment
AbhitejJohn
left a comment
Collaborator
There was a problem hiding this comment.
Curious, how this would work once we move over to Vally. Could we evaluate before we merge - the move to Vally isn't far away and I'd like for these checks to continue once we switch as well.
| /// advisory messages for human review. Entries matching the allowlist are | ||
| /// skipped. | ||
| /// </summary> | ||
| public static IReadOnlyList<string> CheckAgentToolPortability(AgentInfo agent, IReadOnlySet<string>? allowed = null) |
Collaborator
There was a problem hiding this comment.
We'd want to figure out how this would work with Vally. We are close to moving over.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
#847 added
tools: ["agent", "skill", "read", "search", "edit", "execute"]to thecode-testing-*agents to enable subagent fan-out. As that PR notes, those lowercase aliases are portable across VS Code and the Copilot CLI.They are not portable to the other CLI hosts that load these same agents from a marketplace/
--plugin-dir:tools:againstTask,Skill,Read,Glob,Grep,Edit,Write,Bash— and treats the list as an explicit, case-sensitive allow-list. None of the lowercase aliases match, so the agent is granted zero tools. A tool-less model asked to generate tests emits a textual<tool_call>block and exits after one turn with an empty diff. This surfaced ascode-testing-generatorscoring 0 on a Claude-Code-hosted benchmark run ("no changes detected").read_file,replace,write_file,glob,grep_search,run_shell_command.Fix
Make each agent's
tools:list a union of all three hosts' spellings, so one declaration works everywhere — each host honors the names it recognizes and ignores the rest (verified for Claude Code; Gemini/Copilot ignore unknown names too).readReadread_fileedit/createEdit/Writereplace/write_filesearchGlob,Grepglob,grep_searchexecuteBashrun_shell_commandagentTask@agent(no tools entry)skillSkillThe additions are purely to the
tools:frontmatter line of the 8 code-testing agents — no prose or behavior changes — so #847's fan-out is preserved.skill-validator changes (prevention + review feedback)
1. Complete the validator's
BuiltInToolsset. The Copilot review flaggedWriteas non-built-in. Rather than temporaryallowed-external-deps.txtentries (that file is meant to trend to empty), I added the host tool spellings that are legitimate built-ins but weren't case-insensitive matches of existing entries:write,agent,execute(the last two were already flagged pre-branch, from #847), plus the Gemini namesread_file,replace,write_file,grep_search,run_shell_command.2. Add a cross-host tool portability check (
CheckAgentToolPortability). An agent that declares a capability for only one host is now flagged, with an actionable message ("addReadfor Claude Code;read_filefor Gemini CLI"). Names are matched case-sensitively (hosts resolve tools by exact spelling), so the exact bug this PR fixes is caught going forward. The capability table is per-host and generalizes to N hosts; capabilities with notools:-level spelling on a host (Gemini subagents/skills) are not demanded there. Findings are advisory (they do not fail CI) and allowlistable viaagent-tool-portability:AGENT:capability.I also made the one pre-existing single-host agent (
optimizing-dotnet-performance) portable so the check reports a clean tree.Validation
skill-validatorbuilds with 0 warnings; 32ExternalDependencyCheckerTestspass.skill-validator checkacross all 16 plugins passes with 0 findings.