Skip to content

Add agent-audit skill for Cortex Agent best-practices auditing #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Resheras wants to merge 1 commit into
base: main
Choose a base branch
from
+327 −0
Open

Add agent-audit skill for Cortex Agent best-practices auditing #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
184 changes: 184 additions & 0 deletions skills/agent-audit/LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,184 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/

TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION

1. Definitions.

"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.

"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.

"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.

"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.

"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.

"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.

"Work" shall mean the work of authorship made available under
the License, as indicated by a copyright notice that is included in
or attached to the work (an example is provided in the Appendix below).

"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other transformations
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.

"Contribution" shall mean, as submitted to the Licensor for inclusion
in the Work by the copyright owner or by an individual or Legal Entity
authorized to submit on behalf of the copyright owner. For the purposes
of this definition, "submitted" means any form of electronic, verbal,
or written communication sent to the Licensor or its representatives,
including but not limited to communication on electronic mailing lists,
source code control systems, and issue tracking systems that are managed
by, or on behalf of, the Licensor for the purpose of discussing and
improving the Work, but excluding communication that is conspicuously
marked or otherwise designated in writing by the copyright owner as
"Not a Contribution."

"Contributor" shall mean Licensor and any Legal Entity on behalf of
whom a Contribution has been received by the Licensor and subsequently
incorporated within the Work.

2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.

3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a cross-claim
or counterclaim in a lawsuit) alleging that the Work or any
Contribution embodied within the Work constitutes direct or contributory
patent infringement, then any patent licenses granted to You under
this License for that Work shall terminate as of the date such
litigation is filed.

4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:

(a) You must give any other recipients of the Work or Derivative
Works a copy of this License; and

(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and

(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and

(d) If the Work includes a "NOTICE" text file as part of its
distribution, you must include a readable copy of the
attribution notices contained within such NOTICE file, in
at least one of the following places: within a NOTICE text
file distributed as part of the Derivative Works; within
the Source form or documentation, if provided along with the
Derivative Works; or, within a display generated by the
Derivative Works, if and wherever such third-party notices
normally appear. The contents of the NOTICE file are for
informational purposes only and do not modify the License.
You may add Your own attribution notices within Derivative
Works that You distribute, alongside or as an addendum to
the NOTICE text from the Work, provided that such additional
attribution notices cannot be construed as modifying the License.

You may add Your own license statement for Your modifications and
may provide additional grant of rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of the
Contribution, either on an unmodified basis, with modifications,
or as part of a larger work.

5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.

6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.

7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.

8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or exemplary damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or all other
commercial damages or losses), even if such Contributor has been
advised of the possibility of such damages.

9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

END OF TERMS AND CONDITIONS

Copyright 2026 Anastasiia Stefanska

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
143 changes: 143 additions & 0 deletions skills/agent-audit/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
---
id: agent-audit
name: agent-audit
skill-name: $agent-audit
description: Audit and proofread a Snowflake Cortex Agent's configuration against a 22-point best-practices checklist covering correctness, instruction architecture, behavioral controls, and response quality. Use when the user asks to review, audit, proofread, or improve an agent.
prompt: "$agent-audit Review my Cortex Agent for best practices"
language: en
status: Published
author: Anastasiia Stefanska
type: community
---

# Agent Audit

# When to Use
- User asks to review, audit, proofread, or improve a Cortex Agent
- User wants to check their agent's configuration (instructions, orchestration, tool descriptions, tool resources, or profile)
- Do NOT use for building a new agent from scratch

# What This Skill Provides
A comprehensive 22-point audit of a Cortex Agent's full `agent_spec`, grouped into six categories: Correctness & Hygiene, Instruction Architecture, Audience & Domain Context, Behavioral Controls, Response Quality, and Infrastructure & Robustness. Includes an optional Semantic View quality check for agents using a Cortex Analyst tool.

# Instructions

## Step 1: Extract the Full Agent Spec

**Actions:**
1. **Run** `DESCRIBE AGENT <fully_qualified_agent_name>` to retrieve the full agent_spec
2. **If truncated**, extract in chunks using `SUBSTR($7, <offset>, 1000)` from `TABLE(RESULT_SCAN(LAST_QUERY_ID()))` until the full spec is captured

**Output:** Complete agent_spec text ready for audit

**⚠️ STOPPING POINT:** Confirm the full spec is retrieved before proceeding.

## Step 2: Run the 22-Point Checklist

**Actions:**
1. **Evaluate** every section of the agent_spec against all 22 checks below
2. **Record** each finding with its check number and severity

### Correctness & Hygiene
1. **Typos & Spelling** — Scan all instruction text, profile display_name, tool names, and tool_resources for misspellings
2. **Orphan/Placeholder Text** — Look for dangling strings, leftover TODO markers, or incomplete sentences ending with "..."
3. **Valid Tool Resource References** — Verify semantic_view names, search_service names, and warehouse fields are non-empty and plausibly correct
4. **Formatting Consistency** — Check for double spaces, inconsistent newlines, mixed bullet styles, or broken markdown

### Instruction Architecture
5. **No Cross-Layer Duplication** — Response instructions own FORMATTING; orchestration owns ROUTING/LOGIC; tool descriptions own CAPABILITY SCOPE. Flag any rule that appears in more than one layer.
6. **Guardrails Positioned First** — Off-topic refusal, scope boundaries, and safety rules must appear at the TOP of orchestration instructions, not buried in the middle or end
7. **Consolidated "Do Not" Rules** — All negative constraints must live in ONE authoritative list, not scattered across layers

### Audience & Domain Context
8. **Audience Definition Present** — Orchestration must explicitly name primary users and state what they value most
9. **Domain Context Present** — Instructions must name the specific business domain, data domains, or organization
10. **Numbers-in-Context Rule** — Every metric must include denominator/sample context (e.g., "67% based on 2 wins out of 3 closed deals," not just "67%")

### Behavioral Controls
11. **Result Count Cap** — Instructions must explicitly limit how many results are displayed (e.g., "Return at most 5 results unless the user requests more")
12. **Explicit Tool Priority** — Orchestration must state tool selection order clearly (e.g., "Always use Analyst first for quantitative questions")
13. **Keyword Signal Lists** — Orchestration must list trigger words mapped to each tool (quantitative signals → Analyst; qualitative signals → Search)
14. **Sequential Multi-Tool Pattern** — For questions filtering unstructured data by structured criteria, orchestration must enforce query-then-search: first query Analyst for record identifiers, then pass those into the Search query
15. **Empty-Results Handling** — If a structured query returns no data, the agent must report that gap honestly and NOT silently fall back to search
16. **Graceful Edge Cases** — Explicit handling must exist for: no results found, ambiguous queries, and multiple possible interpretations

### Response Quality
17. **Response Format Guidance** — Response instructions must specify structure: lead with a direct answer first, then supporting details; prose over bullet points
18. **Concrete Citation Format** — Require specific citations ("In the TechCorp discovery call"), not vague ones ("according to the data")
19. **Number Formatting Rules** — Standardize number display (e.g., deal values in K suffix: $90,000 → 90K; percentages rounded to whole numbers)
20. **Data Limitation Transparency** — When synthesizing fewer than three data points, the agent must note this explicitly (e.g., "based on two closed Enterprise deals")

### Infrastructure & Robustness
21. **Pinned Orchestration Model** — `models.orchestration` must be a specific model name, not `"auto"`. Flag `"auto"` as a risk.
22. **Tuned max_results** — Values below 5 may be too restrictive; values above 10 may flood context. Recommend 5–6 for most catalog use cases.

**Output:** Full list of findings with check number and severity

**⚠️ STOPPING POINT:** Present findings to the user before suggesting any fixes.

## Step 3: Report Findings by Severity

**Actions:**
1. **Group** all findings into three severity tiers and present as a table:
- **Critical** — Will cause incorrect behavior, silent failures, or broken tool calls (e.g., typo in tool_resource name, missing tool priority, no guardrails)
- **High** — Will cause poor user experience or unreliable responses (e.g., no audience definition, missing citation format, no sequential multi-tool pattern)
- **Nice-to-have** — Polish and robustness improvements (e.g., keyword signal lists, number formatting rules, result count cap)
2. **Confirm** if all 22 checks pass: output "All 22 best-practice checks passed."

**Output:** Severity-grouped findings table

## Step 4: Semantic View Quality Checks (if applicable)

**Actions:** If the agent uses a Cortex Analyst tool backed by a semantic view, also:
1. **Check** synonyms are defined on all key dimensions (customer name, sales stage, sales rep, product line)
2. **Check** custom measures are present for any KPI requiring a formula (e.g., win rate = SUM(CASE WHEN win_status = true THEN 1 ELSE 0 END) / COUNT(*))
3. **Check** sample values are enabled on the semantic view so Cortex Analyst can infer data ranges
4. **Check** AI-generated descriptions are enabled on tables and columns to reduce query planning ambiguity

**Output:** Semantic view findings appended to the main report

## Best Practices
- Always extract the FULL agent_spec before auditing — truncated specs will miss issues
- Pay special attention to tool_resources: typos there cause silent failures
- Check the profile section (display_name, comment) as part of the audit
- Non-determinism is normal — if a use case requires consistent behavior, recommend more detailed orchestration instructions with explicit step-by-step patterns

## Instruction Layer Quick-Reference

Use this table to decide where a rule belongs before auditing or writing instructions.

| Layer | Owns | Does NOT own |
|---|---|---|
| **Orchestration instructions** | Tool selection logic, keyword signals, sequential patterns, scope boundaries, guardrails, audience definition | Output formatting, citation style, number formatting |
| **Response instructions** | Output format, tone, citation style, number formatting, data limitation transparency | Routing logic, tool selection, guardrails |
| **Tool descriptions** | What data the tool contains, when it is appropriate to use | Formatting rules, routing hierarchies that span multiple tools |

## Iteration Loop (Trace-Driven Improvement)

After testing the agent, use this loop to fix issues systematically:

1. **Identify symptom** — What did the agent do wrong? (wrong tool, bad format, speculation, vague citation)
2. **Diagnose the layer** — Wrong tool selection → fix orchestration. Poor response format → fix response instructions. Scope violation → fix boundary rules.
3. **Write a specific fix** — Include a pattern, a rationale, and an example; vague instructions produce inconsistent behavior
4. **Retest via trace** — Confirm the planning span reflects the intended tool call sequence and the response span reflects the format rules
5. **Document the pattern** — Keep a log of issues and fixes; over time this becomes a reusable instruction library

# Stopping Points
- ✋ After Step 1 — confirm full spec is retrieved before running checks
- ✋ After Step 2 — present all findings to the user before suggesting fixes

**Resume rule:** Upon user approval, proceed directly to the next step without re-asking.

# Output
A severity-grouped audit report (Critical / High / Nice-to-have) covering all 22 checks, plus optional Semantic View quality findings. If all checks pass, outputs: "All 22 best-practice checks passed."

# Examples

## Example 1: Basic audit
User: `$agent-audit Review my Cortex Agent for best practices`
Assistant: Prompts for the agent name, runs `DESCRIBE AGENT <name>`, extracts the full spec, evaluates all 22 checks, and returns a grouped findings table.

## Example 2: Audit with a Cortex Analyst tool
User: `$agent-audit Audit my sales agent including the semantic view`
Assistant: Runs all 22 checks plus the four Semantic View Quality Checks, and reports all findings grouped by severity.