Add eks-upgrade-readiness skill

[LearningNewbie]

Description

Adds a new EKS Upgrade Readiness skill (skills/eks-upgrade-readiness) that performs comprehensive pre-upgrade assessments aligned with the AWS EKS Best Practices Guide — Cluster Upgrades.

What It Does

An 11-step upgrade readiness workflow:

1. Cluster context gathering
2. Infrastructure prerequisites (subnet IPs, IAM role, KMS key)
3. EKS Upgrade Insights check (UPGRADE_READINESS category)
4. API deprecation analysis (incl. Dockershim, PSP, in-tree storage removals)
5. Addon compatibility validation with upgrade order
6. Node group readiness (managed, Karpenter Drift/expiry, Cluster Autoscaler, Auto Mode, self-managed)
7. PDB and TopologySpreadConstraints validation
8. Capacity planning (surge calculation, FDCR/ODCR, blue-green alternative)
9. Fargate pod restart considerations
10. Upgrade plan generation with rollback gates and rollback matrix
11. Structured readiness report (READY / NOT READY / READY WITH WARNINGS)

Key Features

• Strictly read-only — only uses describe*, list*, get* APIs. Upgrade actions are recommendations for human approval, never executed.
• Full EKS Best Practices alignment — covers every item in the cluster-upgrades section
• Reference docs — API deprecation schedule, addon version matrix, capacity planning (FDCR/ODCR), troubleshooting (tools, feature removals, blue-green strategy)
• Detection tools referenced — kubent, pluto, kubectl-convert, eksup, GoNoGo, DDS

Type of change

• New skill
• New custom agent
• Update to an existing skill or agent
• Documentation or infrastructure change

License confirmation

• By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0.

[shyamkulkarni]

Thanks for adding this proactive EKS upgrade-readiness skill. The overall direction is valuable, but I recommend addressing the following before treating the result as an authoritative READY/NOT READY decision:

1. Use the EKS Upgrade Insights API as the primary deprecated-API signal. Call ListInsights with category=UPGRADE_READINESS for the exact target Kubernetes version, then call DescribeInsight for each result to collect status, refresh/transition timestamps, affected resources, and recommended actions. Missing, stale, inaccessible, or incomplete insight data must produce UNKNOWN, not PASS. A PASSING insight should be reported as “no incompatible usage observed by EKS Upgrade Insights,” not proof that no deprecated manifests exist. Supplemental manifest and active Helm-release scanning should cover dormant workloads and rollback artifacts.

2. Inventory the complete data plane for version skew. ListNodegroups alone is insufficient. Use DescribeNodegroup, Kubernetes Node kubelet versions, and inventory self-managed ASGs, Karpenter/Auto Mode nodes, and Fargate profiles. Explicitly verify that no node is newer than the control plane and that every node remains within supported skew after the target upgrade.

3. Derive add-on compatibility from live EKS APIs. Use DescribeAddon for current version, health, status, and configuration, and DescribeAddonVersions for source and target cluster versions. Detect self-managed VPC CNI, CoreDNS, and kube-proxy rather than relying only on ListAddons. The static version table should be a dated fallback, not the compatibility authority.

4. Expand AL2 to AL2023 checks beyond amiType/amiFamily. Inspect managed and self-managed launch templates, custom AMI IDs, and decoded user data for /etc/eks/bootstrap.sh, custom kubelet flags, package/systemd hooks, and multipart MIME assumptions. Validate the AL2023 nodeadm/NodeConfig path, cluster metadata, cgroup v2 and IMDSv2 considerations, and recommend a canary replacement node group before migration.

5. Keep assessment read-only. The current read-only claim conflicts with remediation examples such as helm upgrade, blanket rollout restarts, add-on updates with OVERWRITE, PDB patches, node deletion, and ASG changes. Move mutations into a separate optional remediation section with explicit operator approval, backups, scoped targets, and rollback steps.

6. Refine upgrade ordering. Avoid a universal “control plane first” rule. Karpenter, Cluster Autoscaler, admission webhooks, AWS Load Balancer Controller, and similar components may need a version compatible with both source and target before the control-plane update. Distinguish pre-upgrade compatibility remediation from post-upgrade version alignment.

7. Strengthen tests and verdict semantics. Add deterministic cases for N-2/N-3 boundaries, mixed managed/self-managed/Karpenter/Fargate fleets, missing or stale Upgrade Insights, custom AL2 bootstrap user data, managed versus self-managed add-ons, CoreDNS/VPC CNI custom configuration, pagination, and partial AccessDenied. The overall verdict must not be READY while any required gate is UNKNOWN.

Minor corrections: evaluate only the deployed Helm revision as an active blocker; make CRD checks vendor/version-aware; correct the StatefulSet/PVC and minReadySeconds claims; use the actual managed-node-group update algorithm for capacity; document Kubernetes RBAC and all AWS permissions; fix the README archive name and trailing whitespace.

[shyamkulkarni]

Follow-up with additional coverage areas. Credit where due first: the skill already covers PDBs/topology spread, subnet IP availability, audit logging, Velero backups, in-tree-to-CSI migration, webhook-blocked upgrades, extended-support cost, platform version, and blue-green strategy — that's a solid base.

Auditing against a full upgrade-readiness domain checklist, the following areas have no coverage yet. The first five directly affect verdict correctness and would be worth adding in this PR; the rest could be tracked as a follow-up issue.

Recommended for this PR:

1. GitOps/IaC version ownership. If cluster or add-on versions are managed by Terraform, CloudFormation, eksctl, or ArgoCD/Flux, CLI-based remediation creates drift that the next apply reverts. Detect ownership (for example aws:cloudformation:stack-name tags, ArgoCD annotations) and route remediation guidance through the owning tool.

2. VPC CNI networking modes. IPv6 clusters, custom networking, prefix delegation, and security groups for pods each change IP/ENI consumption during node surge and have their own upgrade constraints. The capacity and add-on gates should be mode-aware.

3. Autoscaler behavior during rotation. Recommend pausing Karpenter consolidation and Cluster Autoscaler scale-down during node rotation; otherwise the autoscaler fights the upgrade and churns workloads twice.

4. Pre-upgrade cluster health baseline. Verify all nodes are Ready, no pending CSRs, no crash-looping system pods, and no degraded add-ons before starting. Upgrading an already-degraded cluster is a common source of incidents attributed to the upgrade itself.

5. Post-upgrade functional validation. Pair the READY verdict with a smoke-test workflow: DNS resolution, metrics-server/HPA data flowing, sample workload scheduling, load balancer target health, and IRSA/Pod Identity token exchange.

Candidate follow-up issue:

6. EKS Pod Identity — the Pod Identity Agent add-on has its own version compatibility, and associations must be recreated in a blue-green migration (unlike IRSA trust policies).
7. Accelerated/specialty compute — GPU (NVIDIA driver + device plugin) and Neuron AMI transitions to AL2023 accelerated variants; Windows node upgrade specifics beyond the amiType mention.
8. Quorum-aware stateful drain ordering — Kafka/ZooKeeper/etcd-operator/Elasticsearch need one-at-a-time rotation and drain-order awareness beyond generic PDB checks.
9. Client and CI tooling skew — kubectl (±1 minor), eksctl, Helm, and Terraform provider minimums for the target version.
10. Alternate CNIs and service meshes — detect Cilium/Calico and adjust the add-on gates accordingly instead of assuming VPC CNI.
11. Hybrid nodes and Auto Mode depth — both have distinct skew and upgrade semantics; at minimum detect and flag as requiring separate assessment.
12. Machine-readable output — emit a structured JSON verdict (gates, evidence, timestamps) so the readiness check can gate CI/CD pipelines and be tracked across the proactive cadence this PR targets.
13. Fleet consistency — compare readiness across clusters/regions and flag version sprawl for organizations running the proactive cadence at scale.

Happy to help split these into tracked issues if that's the preferred route.