Fix removeUntaggedEc2: validate billing tags for EKS instances

Previously, EKS clusters without billing tags were deleted if they didn't
match the skip pattern, even if they had valid billing tags. This fix adds
proper billing tag validation for EKS instances.

Changes:
- Add has_valid_billing_tag() to validate category strings and timestamps
- Check billing tags BEFORE skip pattern for EKS instances
- Support Unix timestamp expiration in iit-billing-tag
- Add comprehensive test suite with 12 test scenarios
- All tests pass with proper timezone handling (UTC)

Author: nogueiraanderson

cloud/aws-functions/README_removeUntaggedEc2.md

@@ -0,0 +1,134 @@
+# removeUntaggedEc2 Lambda - Fixed Logic
+
+## Problem
+
+The original Lambda had a bug in how it handled EKS instances:
+
+**OLD BEHAVIOR:**
+- EKS cluster doesn't match `pe-.*` pattern → **Always delete cluster** (ignored `iit-billing-tag`)
+
+**NEW BEHAVIOR:**
+- EKS cluster has valid `iit-billing-tag` (category or future timestamp) → **Keep it**
+- EKS cluster matches `pe-.*` pattern → **Keep it** (protected cluster)
+- EKS cluster has **neither** valid billing tag **nor** matches pattern → **Delete cluster**
+
+## Changes Made
+
+### 1. New Function: `has_valid_billing_tag(tags_dict, instance_launch_time)`
+
+Validates `iit-billing-tag` values:
+- **Category strings** (e.g., `"pmm-staging"`, `"jenkins-pmm-slave"`) → Valid
+- **Unix timestamps** (e.g., `"1759837138"`) → Valid only if timestamp is in the future
+- **Empty or missing** → Invalid
+
+### 2. Updated Function: `is_eks_managed_instance(instance, region)`
+
+New logic flow:
+1. Check if instance has EKS tags
+2. **First check:** Does instance have valid billing tag? → Skip (keep it)
+3. **Second check:** Does cluster name match skip pattern? → Skip (protected)
+4. **Neither:** Mark cluster for deletion
+
+### 3. Updated Function: `is_instance_to_terminate(instance)`
+
+Now uses `has_valid_billing_tag()` for consistent validation.
+
+## Test Coverage
+
+Run `test_removeUntaggedEc2_logic.py` to validate logic without AWS:
+
+```bash
+python3 cloud/aws-functions/test_removeUntaggedEc2_logic.py
+```
+
+### Test Scenarios
+
+**Regular EC2 Instances:**
+- ✓ With category tag (`jenkins-pmm-slave`) → Kept
+- ✓ With future timestamp → Kept
+- ✗ With expired timestamp → Terminated
+- ✗ No tag, running > 10min → Terminated
+- ○ No tag, running < 10min → Kept (grace period)
+
+**EKS Instances - Protected Cluster (`pe-*`):**
+- ✓ `pe-crossplane` without tag → Kept (matches skip pattern)
+- ✓ `pe-infra` with tag → Kept (has billing tag)
+
+**EKS Instances - Non-Protected Cluster:**
+- ✓ `pmm-test` with category tag → Kept (has valid tag)
+- ✓ `pmm-temp` with future timestamp → Kept (has valid tag)
+- ✗ `pmm-ha` without tag → **Cluster deleted**
+- ✗ `pmm-expired` with expired timestamp → **Cluster deleted**
+
+## Deployment
+
+### Option 1: Direct Lambda Update
+
+```bash
+cd cloud/aws-functions
+zip removeUntaggedEc2.zip removeUntaggedEc2.py
+
+aws lambda update-function-code \
+  --function-name removeUntaggedEc2 \
+  --zip-file fileb://removeUntaggedEc2.zip \
+  --region eu-west-1 \
+  --profile percona-dev-admin
+```
+
+### Option 2: Update CloudFormation Stack
+
+If the Lambda is managed by CloudFormation, update the inline code in the stack template.
+
+## Environment Variables
+
+- `EKS_SKIP_PATTERN` (default: `"pe-.*"`) - Regex pattern for protected EKS clusters
+
+## How to Tag EKS Clusters
+
+### Via eksctl Config
+
+```yaml
+apiVersion: eksctl.io/v1alpha5
+kind: ClusterConfig
+metadata:
+  name: pmm-ha
+  region: us-east-2
+  tags:
+    iit-billing-tag: "pmm-eks"
+managedNodeGroups:
+  - name: system
+    tags:
+      iit-billing-tag: "pmm-eks"
+    propagateASGTags: true  # Ensures tags go to EC2 instances
+```
+
+### Via eksctl CLI
+
+```bash
+eksctl create cluster --name pmm-ha \
+  --tags iit-billing-tag=pmm-eks \
+  --managed \
+  --node-labels iit-billing-tag=pmm-eks
+```
+
+### For Existing Clusters
+
+Tag the node groups and update launch templates to propagate tags to new instances.
+
+## Answer to Alex's Question
+
+> Why are we skipping EKS instances if they don't have `iit-billing-tag` set?
+
+**We're not anymore!** The fixed logic now:
+
+1. **Checks billing tag first** - If EKS instance has valid `iit-billing-tag`, it's kept regardless of cluster name
+2. **Then checks skip pattern** - Only if no billing tag, check if cluster matches `pe-.*`
+3. **Deletes if neither** - Cluster gets deleted only if it has no valid billing tag AND doesn't match the pattern
+
+The skip pattern (`pe-.*`) is a whitelist for Platform Engineering's long-running clusters that may not have billing tags but should be protected.
+
+## Files
+
+- `removeUntaggedEc2.py` - Updated Lambda function
+- `test_removeUntaggedEc2_logic.py` - Validation script with 12 test scenarios
+- `README_removeUntaggedEc2.md` - This documentation