Skip to content

[DO NOT MERGE] Test gmp-migrate SKILL.md - #2070

Closed
karthunni wants to merge 5 commits into
karthunni/po-migrate-skillfrom
po-migrate-skill-eval
Closed

[DO NOT MERGE] Test gmp-migrate SKILL.md#2070
karthunni wants to merge 5 commits into
karthunni/po-migrate-skillfrom
po-migrate-skill-eval

Conversation

@karthunni

Copy link
Copy Markdown
Collaborator

[DO NOT MERGE] Agent Skill Verification & Evaluation Harness

self-contained + temporary evaluation harness under hack/migration-eval/ to provide empirical evidence that the gmp-migrate agent skill (cmd/gmp-migrate/SKILL.md) correctly guides an AI agent to reconcile migrated manifests across both positive resolution paths and negative safety guardrails.

These evaluation files are for review verification only and will not be merged into production branches.


To verify that an AI agent using cmd/gmp-migrate/SKILL.md reliably reconciles converted manifests, we test the agent across an isolated evaluation harness:

  1. What We Run:
    • ./hack/migration-eval/setup.sh spins up a local Kind cluster (gmp-eval) and deploys test workloads across 13 isolated namespaces (eval-p1..eval-n4).
    • The agent is provided with test monitor manifests from hack/migration-eval/monitors/
  2. What We Evaluate:
    • Positive Tests (P1P9): Evaluates autonomous reconciliation against Golden YAMLs (hack/migration-eval/golden/) with all TODO annotations cleanly resolved.
    • Negative Tests (N1N4): Evaluates safety guardrails (verifying the agent pauses and asks the human rather than guessing ambiguous ports, fabricating fake credentials, or making arbitrary business decisions).
  3. What We Output:
    • Reconciled manifests verified with kubectl apply --dry-run=server.
    • Execution trajectory logs recording the agent's diagnostics, prompts, and diffs.

@karthunni karthunni self-assigned this Aug 16, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a migration evaluation harness for Google Cloud Managed Service for Prometheus (GMP), adding setup/cleanup scripts, various Prometheus Operator monitors, golden files, and isolated workloads. Feedback on these changes identifies a few critical issues: the eval-n3 namespace definition and its corresponding workload deployment are missing from workloads.yaml, and the secret in offline_corrupt_secret.yaml uses an incorrect namespace and invalid base64 data which will fail API validation. Additionally, it is recommended to add a pre-flight check for the kubectl CLI in setup.sh.

Comment thread hack/migration-eval/monitors/offline_corrupt_secret.yaml Outdated
Comment thread hack/migration-eval/workloads.yaml
Comment thread hack/migration-eval/workloads.yaml
Comment thread hack/migration-eval/setup.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant