[DO NOT MERGE] Test gmp-migrate SKILL.md - #2070
Closed
karthunni wants to merge 5 commits into
Closed
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a migration evaluation harness for Google Cloud Managed Service for Prometheus (GMP), adding setup/cleanup scripts, various Prometheus Operator monitors, golden files, and isolated workloads. Feedback on these changes identifies a few critical issues: the eval-n3 namespace definition and its corresponding workload deployment are missing from workloads.yaml, and the secret in offline_corrupt_secret.yaml uses an incorrect namespace and invalid base64 data which will fail API validation. Additionally, it is recommended to add a pre-flight check for the kubectl CLI in setup.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
[DO NOT MERGE] Agent Skill Verification & Evaluation Harness
self-contained + temporary evaluation harness under
hack/migration-eval/to provide empirical evidence that thegmp-migrateagent skill (cmd/gmp-migrate/SKILL.md) correctly guides an AI agent to reconcile migrated manifests across both positive resolution paths and negative safety guardrails.These evaluation files are for review verification only and will not be merged into production branches.
To verify that an AI agent using
cmd/gmp-migrate/SKILL.mdreliably reconciles converted manifests, we test the agent across an isolated evaluation harness:./hack/migration-eval/setup.shspins up a local Kind cluster (gmp-eval) and deploys test workloads across 13 isolated namespaces (eval-p1..eval-n4).hack/migration-eval/monitors/P1–P9): Evaluates autonomous reconciliation against Golden YAMLs (hack/migration-eval/golden/) with all TODO annotations cleanly resolved.N1–N4): Evaluates safety guardrails (verifying the agent pauses and asks the human rather than guessing ambiguous ports, fabricating fake credentials, or making arbitrary business decisions).kubectl apply --dry-run=server.