Add eval harness for testing AGENTS.md changes#69308

Merged

jason810496 merged 16 commits into

apache:mainfrom

RoyLee1224:feat/skill-eval-harness

Jul 13, 2026

Commits on Jul 12, 2026

Add skill-eval harness scaffold with promptfoo

RoyLee1224
authored and
jason810496
committed
ci: Add prek hook to remind eval on AGENTS.md and SKILL.md changes

RoyLee1224
authored and
jason810496
committed
ci: add OAuth auth and runtime config generation to skill-eval harness

RoyLee1224
authored and
jason810496
committed
feat: add skill-eval harness for AGENTS.md regression testing

RoyLee1224
authored and
jason810496
committed
docs: update skill-eval README and use Helm routing as starter case

RoyLee1224
authored and
jason810496
committed
fix: point skill-eval reminder hook at eval.py, not nonexistent eval.sh

RoyLee1224
authored and
jason810496
committed
refactor: skip working arm when unchanged, show model in output

RoyLee1224
authored and
jason810496
committed
refactor: skip working arm when unchanged, show model, use promptfoo@latest

RoyLee1224
authored and
jason810496
committed
refactor: replace command-routing cases with newsfragment cases from real PRs

RoyLee1224
authored and
jason810496
committed
fix: harden skill-eval harness and add prek hook tests

RoyLee1224
authored and
jason810496
committed
feat: replace eval reminder hook with hash-based proof gate

RoyLee1224
authored and
jason810496
committed
feat: run skill-eval via prek-managed node env, guard partial runs

RoyLee1224
authored and
jason810496
committed
feat: follow the files/ output convention for eval results

RoyLee1224
authored and
jason810496
committed
refactor: build promptfoo config as a dict, serialize to JSON

RoyLee1224
authored and
jason810496
committed
refactor: Make prek the only entry point for the skill-eval

RoyLee1224
authored and
jason810496
committed
Add positive newsfragment cases and view-skill-eval hook

RoyLee1224
authored and
jason810496
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add eval harness for testing AGENTS.md changes#69308

Add eval harness for testing AGENTS.md changes#69308
jason810496 merged 16 commits into
apache:mainfrom
RoyLee1224:feat/skill-eval-harness

Commits on Jul 12, 2026

Add skill-eval harness scaffold with promptfoo

ci: Add prek hook to remind eval on AGENTS.md and SKILL.md changes

ci: add OAuth auth and runtime config generation to skill-eval harness

feat: add skill-eval harness for AGENTS.md regression testing

docs: update skill-eval README and use Helm routing as starter case

fix: point skill-eval reminder hook at eval.py, not nonexistent eval.sh

refactor: skip working arm when unchanged, show model in output

refactor: skip working arm when unchanged, show model, use promptfoo@latest

refactor: replace command-routing cases with newsfragment cases from real PRs

fix: harden skill-eval harness and add prek hook tests

feat: replace eval reminder hook with hash-based proof gate

feat: run skill-eval via prek-managed node env, guard partial runs

feat: follow the files/ output convention for eval results

refactor: build promptfoo config as a dict, serialize to JSON

refactor: Make prek the only entry point for the skill-eval

Add positive newsfragment cases and view-skill-eval hook