Add eks-upgrade-readiness skill

eks-upgrade-readiness-workspace/trigger_test_queries.md

@@ -0,0 +1,129 @@
+# EKS Upgrade Readiness - Trigger Test Queries
+
+Test these queries with Claude Code to verify the skill activates correctly.
+
+## Expected to TRIGGER the skill (should_trigger: true)
+
+### Test 1: Basic upgrade readiness
+```
+I need to upgrade my EKS cluster from 1.29 to 1.30 in us-west-2. Can you check if we're ready? The cluster is called prod-payments and runs about 200 nodes with Karpenter.
+```
+**Expected:** Skill should activate and attempt to run readiness checks
+
+### Test 2: Extended support deadline
+```
+We're on EKS 1.27 and getting emails about extended support ending soon. What do we need to do before the auto-upgrade kicks in? We have a ton of Helm releases and I'm worried about deprecated APIs.
+```
+**Expected:** Skill should activate and focus on API deprecation + timeline
+
+### Test 3: Post-upgrade troubleshooting
+```
+kubectl is giving me 'unable to recognize' errors after we upgraded to 1.31 last week — some of our CronJobs and HPAs stopped working. Can you help figure out what broke?
+```
+**Expected:** Skill should activate to identify deprecated APIs
+
+### Test 4: Safety validation
+```
+Is it safe to upgrade our staging EKS cluster to 1.32? We use the community ingress-nginx controller and have a bunch of StatefulSets running PostgreSQL. Also running AL2 nodes.
+```
+**Expected:** Skill should activate and check version-specific gates (AL2 availability for 1.32+)
+
+### Test 5: Multi-version jump
+```
+My team wants to jump from EKS 1.28 to 1.31 — is that possible in one shot or do we need to go through each version? And what's the risk if we do blue-green instead?
+```
+**Expected:** Skill should activate and explain one-minor-version-at-a-time rule
+
+### Test 6: IPVS mode check
+```
+Can you check if our kube-proxy is using IPVS mode? We're planning to go to 1.35 next quarter and I heard there might be issues.
+```
+**Expected:** Skill should activate and check version-specific gate (IPVS deprecated in 1.35)
+
+### Test 7: Karpenter AMI behavior
+```
+We run Karpenter v0.32 with EC2NodeClasses pinned to a specific AMI. Will the nodes automatically pick up the new AMI after we upgrade the control plane to 1.30?
+```
+**Expected:** Skill should activate and assess Karpenter configuration
+
+### Test 8: Service quota check
+```
+Can you check our service quotas for EC2 vCPUs and EBS volumes? We're about to do a rolling node group upgrade on a 100-node cluster and I want to make sure we have headroom.
+```
+**Expected:** Skill should activate for capacity planning / quota check
+
+---
+
+## Expected to NOT TRIGGER the skill (should_trigger: false)
+
+### Test 9: API server throttling (general EKS troubleshooting)
+```
+I'm getting 429 throttling errors on my EKS cluster API server. kubectl get pods takes 5 seconds. The cluster is on 1.30 and has about 500 pods.
+```
+**Expected:** Skill should NOT activate (not upgrade-related)
+
+### Test 10: Security review (not upgrade-specific)
+```
+Can you review the security posture of my EKS cluster? I want to check RBAC, pod security standards, network policies, and whether we're using IRSA or Pod Identity correctly.
+```
+**Expected:** Skill should NOT activate (security audit, not upgrade)
+
+### Test 11: Initial Karpenter setup (not upgrade)
+```
+Help me set up Karpenter on my new EKS cluster. I need NodePools for both x86 and Graviton instances with consolidation enabled.
+```
+**Expected:** Skill should NOT activate (initial setup, not upgrade)
+
+### Test 12: OOM troubleshooting (not upgrade-related)
+```
+My pods keep getting OOMKilled on my EKS cluster. The nodes show 85% memory utilization. How do I right-size the resource requests?
+```
+**Expected:** Skill should NOT activate (resource tuning, not upgrade)
+
+### Test 13: ECS to EKS migration (not EKS upgrade)
+```
+I want to migrate my application from ECS to EKS. What's the best approach for a Java Spring Boot app with a PostgreSQL database?
+```
+**Expected:** Skill should NOT activate (migration, not upgrade)
+
+### Test 14: Terraform generation (IaC, not upgrade assessment)
+```
+Can you generate a Terraform module for an EKS cluster with managed node groups, VPC CNI, and CoreDNS? Target version 1.30 in us-east-1.
+```
+**Expected:** Skill should NOT activate (infrastructure provisioning, not upgrade assessment)
+
+### Test 15: Cost optimization (not upgrade-related)
+```
+Our EKS cluster costs are too high — $15K/month. Can you analyze where the money is going and suggest optimizations? We have 50% of nodes running at under 20% CPU.
+```
+**Expected:** Skill should NOT activate (cost analysis, not upgrade)
+
+### Test 16: RDS upgrade (wrong service)
+```
+We need to upgrade our RDS PostgreSQL from 14 to 16. Can you check addon compatibility and if there are any breaking changes?
+```
+**Expected:** Skill should NOT activate (RDS, not EKS)
+
+---
+
+## How to Test
+
+1. Open Claude Code in the terminal or desktop app
+2. Make sure the eks-upgrade-readiness skill is installed
+3. Type one of the test queries above
+4. Observe whether Claude mentions using or consulting the eks-upgrade-readiness skill
+5. For "should trigger" tests: verify the skill activates
+6. For "should NOT trigger" tests: verify Claude answers without the skill
+
+## Pass Criteria
+
+- **Should trigger queries (1-8):** At least 7/8 should activate the skill (87.5% trigger rate)
+- **Should NOT trigger queries (9-16):** All 8 should NOT activate the skill (100% precision)
+- **Functional evals (evals.json):** 16 scenarios covering normal, edge, and failure cases
+- **Deterministic rules:** UNKNOWN ≠ PASS, READY not allowed with UNKNOWN gates
+
+## Notes
+
+- Trigger rate depends on Claude's skill selection logic, not just the description
+- Some edge cases (like query 8 about service quotas) might be borderline
+- The skill description aims for high recall (catch all upgrade-related queries) while excluding non-upgrade EKS work

skills/eks-upgrade-readiness/.skilleval.yaml

@@ -0,0 +1,3 @@
+audit:
+  ignore:
+    - STR-016    # README alongside SKILL.md is intentional

skills/eks-upgrade-readiness/CHANGELOG.md

@@ -0,0 +1,119 @@
+# Changelog
+
+## 2.0.0
+
+Major rewrite addressing PR #48 review feedback. Breaking changes to step
+numbering and report format.
+
+### Must-Fix Items Addressed
+
+1. **EKS Upgrade Insights as primary signal** — Step 3 now explicitly declares
+   Insights as the primary authoritative signal. UNKNOWN verdict (not PASS) when
+   Insights is unavailable or returns no data. Pagination enforced.
+
+2. **Complete data plane inventory** — New Step 6 inventories ALL node
+   populations: Managed Node Groups (with DescribeNodegroup details),
+   self-managed ASGs (via autoscaling API + launch template inspection),
+   Karpenter (NodePools + EC2NodeClasses), Auto Mode, and Fargate profiles.
+   Kubelet version map across all nodes with skew validation.
+
+3. **Live addon API usage** — Step 5 uses `DescribeAddon` + `DescribeAddonVersions`
+   as primary source. Self-managed addon detection via deployment/Helm scan.
+   Static addon-version-matrix.md is now explicitly a fallback-only reference.
+
+4. **AL2→AL2023 comprehensive migration** — New Step 7 covers launch template
+   analysis, custom AMI detection, user data bootstrap differences (bootstrap.sh
+   → nodeadm/NodeConfig), cgroup v2 compatibility, IMDSv2 defaults, yum→dnf.
+
+5. **Mutations separated into Remediation Playbook** — New Step 14 consolidates
+   ALL mutating commands (helm upgrade, rollout restarts, OVERWRITE addon
+   updates, PDB patches, Karpenter annotation, CA pause). Clearly marked as
+   requiring operator approval. Agent never executes these.
+
+6. **Upgrade ordering refined** — New Step 8 explicitly defines pre-upgrade
+   alignment (Karpenter, CA, webhooks may need update BEFORE control plane) vs
+   post-upgrade addon/node-group sequence.
+
+7. **Deterministic test cases** — evals.json expanded from 6 to 16 scenarios
+   covering: N-2/N-3 mixed fleet, version skew violations, missing Insights
+   (AccessDenied), custom bootstrap AL2→AL2023, pagination handling, Karpenter
+   Drift disabled, GitOps/IaC detection, post-upgrade validation, self-managed
+   addon detection, pre-upgrade health failure. Assertions enforce UNKNOWN≠PASS.
+
+### Recommended Additions Addressed
+
+1. **GitOps/IaC version ownership detection** — New Step 12 detects CF/TF/CDK/
+   ArgoCD/Flux/eksctl from cluster/nodegroup tags and routes all remediation
+   through the owning tool. Never suggests direct CLI if IaC-managed.
+
+2. **VPC CNI networking modes awareness** — Step 2 subnet check is now mode-
+   aware: standard, prefix delegation, custom networking, IPv6, Security Groups
+   for Pods. Includes ENIConfig detection and mode-specific capacity gates.
+
+3. **Autoscaler pause during rotation** — New Step 13 checks Karpenter
+   consolidation policy and CA scale-down state. Pause commands in Step 14.
+
+4. **Pre-upgrade cluster health baseline** — New Step 10 validates all nodes
+   Ready, no pending CSRs, no crash-looping system pods, DNS resolution working,
+   metrics-server responding. Any failure blocks the upgrade.
+
+5. **Post-upgrade functional validation** — New Step 15 provides smoke tests:
+   DNS, metrics-server, pod scheduling, LB health, IRSA/Pod Identity, baseline
+   comparison.
+
+### Additional Improvements
+
+- Documented required AWS IAM permissions and Kubernetes RBAC ClusterRole
+- Added AccessDenied handling protocol (mark UNKNOWN, never PASS)
+- Added pagination requirements throughout (ListInsights, ListNodegroups, etc.)
+- Helm scanning now checks deployed revision only (not full history)
+- CRD checks are vendor-aware and version-aware (compare installed version)
+- StatefulSet PVC check corrected (persistentVolumeClaimRetentionPolicy)
+- MNG update algorithm documented (for capacity planning accuracy)
+- Verdict rules formalized: CANNOT DETERMINE when any gate is UNKNOWN
+- Trigger eval expanded to 16 queries (8 positive, 8 negative)
+- Skill expanded from 11 to 17 steps
+- Version bump to 2.0.0
+
+## 1.2.0
+
+- Add Helm manifest scanning — detects deprecated APIs in Helm release
+- Add version-specific removal gates — AL2 AMI unavailability (≥1.33),
+  kube-proxy IPVS deprecation (≥1.35/1.36), unmaintained ingress-nginx
+- Add service quota headroom checks
+- Add StatefulSet safety checks
+- Add more comprehensive Karpenter checks
+- Add scaled-to-zero workload detection
+- Add grading guards with confidence levels
+- Add third-party CRD API deprecation checks
+- Add cost awareness section
+- Add conditional evaluation logic — version-gated checks only when relevant
+- Expand pre-upgrade checklist with new checks
+- Version bump to 1.2.0
+
+## 1.1.0
+
+- Add infrastructure prerequisites check (subnet IPs, IAM role, KMS key)
+- Add Karpenter Drift and node expiry handling
+- Add TopologySpreadConstraints validation
+- Add Fargate pod restart requirement (Step 9)
+- Add feature-specific removals (Dockershim, PodSecurityPolicy, in-tree storage)
+- Add detection tools: kubent, pluto, kubectl-convert, eksup, GoNoGo
+- Add blue-green cluster alternative for large upgrades
+- Add EKS release calendar and auto-upgrade policy context
+- Add EKS Auto Mode awareness
+- Add rollback matrix
+- Expand from 9 steps to 11 steps
+- Align fully with AWS EKS Best Practices Guide cluster-upgrades section
+
+## 1.0.0
+
+- Initial version
+- 9-step upgrade readiness assessment workflow
+- API deprecation analysis with version-specific removal matrix
+- Addon compatibility check against target EKS version
+- Node group version skew and AMI readiness validation
+- Pod Disruption Budget validation for drain safety
+- Capacity planning with surge calculation and reservation guidance
+- Structured upgrade plan generation with rollback gates
+- Reference documents for API deprecations, addon matrix, capacity planning, and troubleshooting