Conversation
Cherry-picked only the /harden-specific files from PR #247 (harden.prompt.md, SKILL.md, instruction-rules.md, check_instruction_budget.py, list_agent_capabilities.py, and their tests) rather than merging the whole branch, since pr-247 has diverged from main with unrelated evaluation-skill changes. Capability/menu/telemetry wiring deferred to a later step.
Ported models.py, normalize.py, parser.py, and regex_rules/{coverage,antipattern}.py
from the standalone ESSAgentInstructionValidator repo into
solutions/ess-maker-skills/scripts/instruction_engine/, along with its existing
test suite adjusted to the new import path. This becomes the deterministic
regex pass that the shared engine (Step 5) will run alongside the semantic
LLM pass ported from PR #247.
Finding now carries source: "regex" | "semantic", defaulting to "regex" so existing regex rule functions need no changes. Schema test confirms a regex-origin and a semantic-origin Finding serialize to the same shape, which the engine (Step 5) relies on to merge both passes' output into one findings list.
Moves instruction-rules.md under instruction_engine/semantic_rules/ and adds runner.py, which makes the INSTR-0xx rule pack invocable outside an interactive skill session. Reuses evaluate_evals.py's existing LLM call convention (GitHub Copilot chat completions API via 'gh auth token' — no new API keys or setup), resolving the design doc's open item on LLM invocation mechanism. run_semantic_pass() takes a pluggable llm_client so tests stub the model call and only verify parsing/schema-conformance. A @pytest.mark.live smoke test exercises a real call and is skipped by default, per the repo's existing --run-live convention.
check_coverage(instructions, reported_problem) runs the semantic pass scoped
to the reported behavior. If the pass surfaces no on-topic
contradiction/gap, an existing rule must already correctly address it and
the verdict is likely_platform_limitation instead of another rewrite
proposal. This is deliberately semantic-pass-only — an earlier version also
fell back to the unscoped regex pass, which produced false-positive
fix_proposed verdicts from keyword overlap with unrelated housekeeping
rules (AP006, COV004) on both Blackstone fixtures.
blackstone_scenario_{1,2}.json capture the real customer data (unsuppressed
'More references' panel despite citation scoping; unsuppressed follow-up
suggestion after a read-only Workday topic despite three prior instruction
attempts) as permanent regression fixtures — both expected to yield
likely_platform_limitation.
run(instructions_text, problems=None) -> EngineResult always runs the regex pass and an unscoped semantic pass, and additionally runs check_coverage for each ReportedProblem in problems, in order.
python -m instruction_engine --instructions FILE [--problems FILE] [--json] [--fail-on error|warn|review] renders the engine.run() result as text or JSON, and exits non-zero when a finding at or above --fail-on severity is present. --problems loads a JSON list of reported-problem records into ReportedProblem for coverage-check triage.
Step 3 now runs `python -m instruction_engine --problems ... --json` against a maker-reported problem before proposing changes. On likely_platform_limitation, the skill is told to say so plainly and route to /flightcheck/escalation instead of proposing another reworded prohibition — this is the whole point of coverage-check mode (real Blackstone data showed re-wording an already-correct rule doesn't help). Also updates the instruction-rules.md path reference from its old location (src/reference/ess-docs/hardening/, moved in Step 3) to scripts/instruction_engine/semantic_rules/, and updates the wiring test accordingly.
Ports the capability taxonomy entry, menu row, and README section from PR #247 that were deliberately deferred when the harden skill scaffolding was cherry-picked in Step 0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Author
@microsoft-github-policy-service agree company="Microsoft" |
Author
@microsoft-github-policy-service agree company="Microsoft" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/hardenskill scaffolding from feat(ess-maker-skills): add /harden instruction review skill #247, adding a unifiedinstruction_enginepackage undersolutions/ess-maker-skills/scripts/that combines two prior efforts:ESSAgentInstructionValidatorINSTR-001..INSTR-033)engine.pyorchestration (run()) and apython -m instruction_engineCLI (text/JSON output,--fail-on,--problemsfor coverage-check triage)/harden(SKILL.md) to call coverage-check before proposing a fix for a reported problem, distinguishing a real instruction gap (fix_proposed) from a platform limitation the skill can't reach by editing instructions (likely_platform_limitation)ADK_CAPABILITIESentry, menu row, and README section for/hardenthat feat(ess-maker-skills): add /harden instruction review skill #247 introducedTest plan
pytest tests/scripts/test_instruction_engine_engine.py tests/scripts/test_instruction_engine_cli.py -q— new engine/CLI tests passpytest tests/scripts/test_instruction_budget.py tests/test_adk_telemetry.py tests/scripts/test_agent_capabilities.py -q— wiring tests passpytest -q— 869 passed, 14 skipped; remaining 14 failures / 37 collection errors confirmed pre-existing onmain(missingdefusedxml/jsonschemadeps, staleonboarding/tasks.mdpath references — unrelated to this change)🤖 Generated with Claude Code