Add GKE cluster for biometric public facing app

[obriensystems]

Prototyping IaC in https://github.com/ObrienlabsDev/gcp-infrastructure-as-code/tree/main/m1-gke-ingress/kubernetes-gke
code for the spring boot app is in https://github.com/ObrienlabsDev/biometric-backend
Temporary app only (no mysql RDB) at https://hub.docker.com/repository/docker/obrienlabs/magellan-nbi/tags

Overview

ops

• https certificate takes up to 45 min to provision, until then the managed certificate k8s resource will not transition from "Provisioning". Once the certificate is ready after the A record in DNS has propagated - the ingress load balancer backend will resolve.
  Look at using https://gateway-api.sigs.k8s.io/

status

20251103:0930 GCP managed certificate working, https endpoint ok
https://magellan.obrienlabs.dev/nbi/swagger-ui.html

details, A record to reserved IP, ingress with defaultBackend - it takes up to 45 min for the GCP managed-certificate to resolve. GKE yamls in
https://github.com/ObrienlabsDev/gcp-infrastructure-as-code/tree/main/m1-gke-ingress/kubernetes-gke

20251030:1900 - GKE ingress through external classic application load balancer routing ok outside - moving on to artifact registry change from dockerhub and terraforming
20251031:0930 - Switched to Artifact Registry - working terraforming
https://dev.azure.com/JDCP/Cloud%20Operations/_workitems/edit/7175

Links

• https://cloud.google.com/build/docs/interacting-with-dockerhub-images

• Add GKE cluster for biometric public facing app blog#141

Requirements

R1: GCP deployment

R2: GKE containerization

R3: Terraform IaC

R4: cloud build triggers

R5: Artifact Registry

• https://docs.cloud.google.com/artifact-registry/docs/docker/store-docker-container-images#add-image

Design Issues

DI1: Ingress Architecture

[obriensystems]

Manual dry run on lenovo p1gen6

  gcloud auth login
  gcloud config set project biometric-backend-gke-man-old

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code (main)
$ gcloud services list
NAME                                 TITLE
analyticshub.googleapis.com          Analytics Hub API
bigquery.googleapis.com              BigQuery API
bigqueryconnection.googleapis.com    BigQuery Connection API
bigquerydatapolicy.googleapis.com    BigQuery Data Policy API
bigquerydatatransfer.googleapis.com  BigQuery Data Transfer API
bigquerymigration.googleapis.com     BigQuery Migration API
bigqueryreservation.googleapis.com   BigQuery Reservation API
bigquerystorage.googleapis.com       BigQuery Storage API
cloudapis.googleapis.com             Google Cloud APIs
cloudtrace.googleapis.com            Cloud Trace API
dataform.googleapis.com              Dataform API
dataplex.googleapis.com              Cloud Dataplex API
datastore.googleapis.com             Cloud Datastore API
logging.googleapis.com               Cloud Logging API
monitoring.googleapis.com            Cloud Monitoring API
servicemanagement.googleapis.com     Service Management API
serviceusage.googleapis.com          Service Usage API
sql-component.googleapis.com         Cloud SQL
storage-api.googleapis.com           Google Cloud Storage JSON API
storage-component.googleapis.com     Cloud Storage
storage.googleapis.com               Cloud Storage API

enable services
gcloud services enable container.googleapis.com

during triage
gcloud services enable compute.googleapis.com



Check roles
roles/billing.admin
roles/resourcemanager.folderCreator
roles/resourcemanager.organizationAdmin
roles/orgpolicy.policyAdmin

GKE creation defaults / overrides

• use default GCE service account
  Cluster basics
  Cluster name: dev
  Cluster location: northamerica-northeast1
  Fleet registration
  Register to a fleet: Disabled
  Networking
  Access using DNS: Enabled
  Access using IPv4 addresses: Enabled
  Enable authorized networks: Disabled
  Access using the control plane's external IP address: Enabled
  Access using the control plane's internal IP address from any region: Enabled
  Enforce authorized networks on the control plane's internal endpoint: Disabled
  Add Google Cloud external IP addresses to authorized networks: Enabled
  Network: default
  Subnetwork: default
  Enable Private Nodes: Enabled
  Override control plane's default private endpoint subnetwork: Disabled
  Cluster default pod address range: /17
  Dataplane V2 Observability: Disabled
  VPC firewall rule auto-creation for LoadBalancer Services: Enabled
  Advanced settings
  Release channel: Stable channel
  Maintenance window: Disabled
  Service Mesh: Disabled
  Backup for GKE: Disabled
  Binary authorization: Disabled
  Secret Manager: Enabled
  Google Groups for RBAC: Disabled
  Secret encryption at the application layer: Disabled
  Security posture: Basic
  Workload vulnerability scanning: Disabled
  Boot disk encryption: Google-managed
  Service account: Default
  Access scopes: Full access
  Ray Operator: Disabled
  Logging: Enabled
  Cloud Monitoring: Enabled
  Automatic Application Metric Collection: Disabled
  Description: gke
  Labels: gke: dev

gcloud beta container --project "biometric-backend-gke-man-old" clusters create-auto "dev" --region "northamerica-northeast1" --release-channel "stable" --enable-private-nodes --enable-dns-access --enable-k8s-tokens-via-dns --enable-k8s-certs-via-dns --enable-ip-access --enable-master-global-access --enable-google-cloud-access --network "projects/biometric-backend-gke-man-old/global/networks/default" --subnetwork "projects/biometric-backend-gke-man-old/regions/northamerica-northeast1/subnetworks/default" --cluster-ipv4-cidr "/17" --binauthz-evaluation-mode=DISABLED --scopes=https://www.googleapis.com/auth/cloud-platform --enable-secret-manager

$ gcloud components update
Beginning update. This process may take several minutes.


Your current Google Cloud CLI version is: 481.0.0
You will be upgraded to version: 545.0.0

+----------------------------------------------------------------------+
|                  These components will be removed.                   |
+-------------------------------+--------------------+-----------------+
|              Name             |      Version       |       Size      |
+-------------------------------+--------------------+-----------------+
| Bundled Python 2.7            |             2.7.13 |                 |
+-------------------------------+--------------------+-----------------+
+-----------------------------------------------------------------------------+
|                      These components will be updated.                      |
+-----------------------------------------------------+------------+----------+
|                         Name                        |  Version   |   Size   |
+-----------------------------------------------------+------------+----------+
| BigQuery Command Line Tool                          |     2.1.25 |  1.8 MiB |
| BigQuery Command Line Tool (Platform Specific)      |     2.1.17 |  < 1 MiB |
| Bundled Python 3.12 (Platform Specific)             |    3.12.10 | 21.2 MiB |
| Cloud Storage Command Line Tool                     |       5.35 | 12.4 MiB |
| Cloud Storage Command Line Tool (Platform Specific) |       5.34 |  < 1 MiB |
| Google Cloud CLI Core Libraries                     | 2025.10.24 | 22.8 MiB |
| Google Cloud CRC32C Hash Tool (Platform Specific)   |      1.0.0 |  1.5 MiB |
| anthoscli (Platform Specific)                       |     0.2.56 | 70.5 MiB |
| gcloud Alpha Commands                               | 2025.10.24 |  < 1 MiB |
| gcloud Beta Commands                                | 2025.10.24 |  < 1 MiB |
| gcloud cli dependencies                             | 2025.10.24 | 16.1 MiB |

$ gcloud beta container --project "biometric-backend-gke-man-old" clusters create-auto "dev" --region "northamerica-northeast1" --release-channel "stable" --enable-private-nodes --enable-dns-access --enable-k8s-tokens-via-dns --enable-k8s-certs-via-dns --enable-ip-access --enable-master-global-access --enable-google-cloud-access --network "projects/biometric-backend-gke-man-old/global/networks/default" --subnetwork "projects/biometric-backend-gke-man-old/regions/northamerica-northeast1/subnetworks/default" --cluster-ipv4-cidr "/17" --binauthz-evaluation-mode=DISABLED --scopes=https://www.googleapis.com/auth/cloud-platform --enable-secret-manager
Note: This cluster has private nodes. If you need connectivity to the public internet, for example to pull public containers, you must configure Cloud NAT. To enable NAT for the network of this cluster, run the following commands:
gcloud compute routers create my-router --region northamerica-northeast1 --network default --project=biometric-backend-gke-man-old
gcloud beta compute routers nats create nat --router=my-router --region=northamerica-northeast1 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges --project=biometric-backend-gke-man-old
ERROR: (gcloud.beta.container.clusters.create-auto) ResponseError: code=400, message=INVALID_CIDR: "C:/soft/Git/17" is not a valid CIDR request, must be /SIZE or IP/SIZE (e.g. '/16', '10.1.0.0/16').

enable compute - to create the VPC default and change to 10.0.0.0/17
 gcloud services enable compute.googleapis.com

$ gcloud beta container --project "biometric-backend-gke-man-old" clusters create-auto "dev" --region "northamerica-northeast1" --release-channel "stable" --enable-private-nodes --enable-dns-access --enable-k8s-tokens-via-dns --enable-k8s-certs-via-dns --enable-ip-access --enable-master-global-access --enable-google-cloud-access --network "projects/biometric-backend-gke-man-old/global/networks/default" --subnetwork "projects/biometric-backend-gke-man-old/regions/northamerica-northeast1/subnetworks/default" --cluster-ipv4-cidr "10.0.0.0/17" --binauthz-evaluation-mode=DISABLED --scopes=https://www.googleapis.com/auth/cloud-platform --enable-secret-manager
Note: This cluster has private nodes. If you need connectivity to the public internet, for example to pull public containers, you must configure Cloud NAT. To enable NAT for the network of this cluster, run the following commands:
gcloud compute routers create my-router --region northamerica-northeast1 --network default --project=biometric-backend-gke-man-old
gcloud beta compute routers nats create nat --router=my-router --region=northamerica-northeast1 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges --project=biometric-backend-gke-man-old
Creating cluster dev in northamerica-northeast1...

10-15 min

expected 30 min timeout - moving from nane1 to us-central-1

.......done.
ERROR: (gcloud.beta.container.clusters.create-auto) Operation [<Operation
 clusterConditions: [<StatusCondition
 canonicalCode: CanonicalCodeValueValuesEnum(UNAVAILABLE, 15)
 code: CodeValueValuesEnum(GCE_STOCKOUT, 1)
 message: "Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'projects/biometric-backend-gke-man-old/zones/northamerica-northeast1-a' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.">, <StatusCondition
 canonicalCode: CanonicalCodeValueValuesEnum(UNAVAILABLE, 15)
 code: CodeValueValuesEnum(GCE_STOCKOUT, 1)
 message: "Google Compute Engine: Not all instances running in IGM after 35m22.071141789s. Expected 1, running 0, transitioning 1. Current errors: [GCE_STOCKOUT]: Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'projects/biometric-backend-gke-man-old/zones/northamerica-northeast1-a' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.">]
 detail: "Google Compute Engine: Not all instances running in IGM after 35m22.071141789s. Expected 1, running 0, transitioning 1. Current errors: [GCE_STOCKOUT]: Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'projects/biometric-backend-gke-man-old/zones/northamerica-northeast1-a' does not have enough resources available to fulfill the request.  Try a different zone, or try again later."
 endTime: '2025-10-28T19:49:02.189708632Z'
 error: <Status
 code: 14
 details: []
 message: "Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'projects/biometric-backend-gke-man-old/zones/northamerica-northeast1-a' does not have enough resources available to fulfill the request.  Try a different zone, or try again later.">
 name: 'operation-1761678459023-86838a5a-240d-498a-8413-de6af0c13bb0'
 nodepoolConditions: []
 operationType: OperationTypeValueValuesEnum(CREATE_CLUSTER, 1)
 progress: <OperationProgress
 metrics: [<Metric
 intValue: 10
 name: 'CLUSTER_CONFIGURING'>, <Metric
 intValue: 10
 name: 'CLUSTER_CONFIGURING_TOTAL'>, <Metric
 intValue: 13
 name: 'CLUSTER_DEPLOYING'>, <Metric
 intValue: 13
 name: 'CLUSTER_DEPLOYING_TOTAL'>]
 stages: []>
 selfLink: 'https://container.googleapis.com/v1beta1/projects/896237704811/locations/northamerica-northeast1/operations/operation-1761678459023-86838a5a-240d-498a-8413-de6af0c13bb0'
 startTime: '2025-10-28T19:07:39.023715112Z'
 status: StatusValueValuesEnum(DONE, 3)
 statusMessage: "Google Compute Engine: Not all instances running in IGM after 35m22.071141789s. Expected 1, running 0, transitioning 1. Current errors: [GCE_STOCKOUT]: Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'projects/biometric-backend-gke-man-old/zones/northamerica-northeast1-a' does not have enough resources available to fulfill the request.  Try a different zone, or try again later."
 targetLink: 'https://container.googleapis.com/v1beta1/projects/896237704811/locations/northamerica-northeast1/clusters/dev'
 zone: 'northamerica-northeast1'>] finished with error: Google Compute Engine: Not all instances running in IGM after 35m22.071141789s. Expected 1, running 0, transitioning 1. Current errors: [GCE_STOCKOUT]: Instance 'gk3-dev-default-pool-8c4748e5-v10k' creation failed: The zone 'proj

20251028:1616

$ gcloud beta container --project "biometric-backend-gke-man-old" clusters create-auto "dev" --region "us-east1" --release-channel "stable" --enable-private-nodes --enable-dns-access --enable-k8s-tokens-via-dns --enable-k8s-certs-via-dns --enable-ip-access --enable-master-global-access --enable-google-cloud-access --network "projects/biometric-backend-gke-man-old/global/networks/default" --subnetwork "projects/biometric-backend-gke-man-old/regions/us-east1/subnetworks/default" --cluster-ipv4-cidr "10.0.0.0/17" --binauthz-evaluation-mode=DISABLED --scopes=https://www.googleapis.com/auth/cloud-platform --enable-secret-manager
Note: This cluster has private nodes. If you need connectivity to the public internet, for example to pull public containers, you must configure Cloud NAT. To enable NAT for the network of this cluster, run the following commands:
gcloud compute routers create my-router --region us-east1 --network default --project=biometric-backend-gke-man-old
gcloud beta compute routers nats create nat --router=my-router --region=us-east1 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges --project=biometric-backend-gke-man-old
Creating cluster dev in us-east1...

[obriensystems]

20251029: Using GKE cluster

• pending: artifact registry, ingress, cloud build

verify docker image local run before moving to artifact registry

We will use a simpler spring boot app for now that does not need a MySQL database

$ docker pull obrienlabs/magellan-nbi:0.0.3-ia64
0.0.3-ia64: Pulling from obrienlabs/magellan-nbi
Digest: sha256:a2d143d836c2b01236fb2fc74e7a3d1949b2c7acf4b6058769daff0639746c12
Status: Image is up to date for obrienlabs/magellan-nbi:0.0.3-ia64
docker.io/obrienlabs/magellan-nbi:0.0.3-ia64

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ docker run -d -p 8888:8080 --name magellan obrienlabs/magellan-nbi:0.0.3-ia64
de8497122d9fdf3c3e3225a260d7736be13d6de16f51e88691094e93b6dd4b27

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS                                         NAMES
de8497122d9f   obrienlabs/magellan-nbi:0.0.3-ia64   "java -Djava.securit…"   30 seconds ago   Up 30 seconds   0.0.0.0:8888->8080/tcp, [::]:8888->8080/tcp   magellan

curl -X GET "http://localhost:8888/nbi/app/health" -H "accept: */*"
ok

Get kubectl access to the GKE instance

$ kubectl version
Client Version: v1.34.1
Kustomize Version: v5.7.1
Unable to connect to the server: getting credentials: exec: executable gke-gcloud-auth-plugin.exe not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

Install gke-gcloud-auth-plugin for use with kubectl by following https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin

follow https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl#install_plugin

$ gke-gcloud-auth-plugin --version
bash: gke-gcloud-auth-plugin: command not found

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ gcloud components install gke-gcloud-auth-plugin


Your current Google Cloud CLI version is: 545.0.0
Installing components from version: 545.0.0

+----------------------------------------------------------------+
|              These components will be installed.               |
+--------------------------------------------+---------+---------+
|                    Name                    | Version |   Size  |
+--------------------------------------------+---------+---------+
| gke-gcloud-auth-plugin (Platform Specific) |  0.5.10 | 3.6 MiB |
+--------------------------------------------+---------+---------+

verify
$ gke-gcloud-auth-plugin --version
Kubernetes v1.30.0+03fcd0f8cb9eac57e97a3ed59c702bad8c73be81

connect to GKE

gcloud container clusters get-credentials dev --location=us-east1
or
$ gcloud container clusters get-credentials dev --region us-east1 --project biometric-backend-gke-man-old --dns-endpoint
Fetching cluster endpoint and auth data.
kubeconfig entry generated for dev.

$ kubectl get namespaces
NAME                           STATUS   AGE
default                        Active   13h
gke-gmp-system                 Active   13h
gke-managed-cim                Active   13h
gke-managed-filestorecsi       Active   13h
gke-managed-parallelstorecsi   Active   13h
gke-managed-system             Active   13h
gke-managed-volumepopulator    Active   13h
gmp-public                     Active   13h
kube-node-lease                Active   13h
kube-public                    Active   13h
kube-system                    Active   13h


however all pods are pending
$ kubectl get pods --all-namespaces
NAMESPACE         NAME                                                       READY   STATUS    RESTARTS   AGE
gke-gmp-system    alertmanager-0                                             0/2     Pending   0          15h
gke-gmp-system    gmp-operator-59bb7c6ddb-tm4vs                              0/1     Pending   0          15h
gke-gmp-system    rule-evaluator-6c5bc8d6dc-v4b42                            0/2     Pending   0          15h
gke-managed-cim   kube-state-metrics-0                                       0/2     Pending   0          15h
kube-system       antrea-controller-horizontal-autoscaler-54745fb775-sf6pb   0/1     Pending   0          15h
kube-system       egress-nat-controller-86c7bc9c6d-x65gv                     0/1     Pending   0          15h
kube-system       event-exporter-gke-78cfd77cbf-rm6fb                        0/2     Pending   0          15h
kube-system       filestore-lock-release-controller-55fb8b59f5-gkt9w         0/2     Pending   0          15h
kube-system       konnectivity-agent-85988db6dd-szr8m                        0/2     Pending   0          15h
kube-system       konnectivity-agent-autoscaler-8567d9c95d-2dskj             0/1     Pending   0          15h
kube-system       kube-dns-594d77f8c4-tcwc9                                  0/4     Pending   0          15h
kube-system       kube-dns-autoscaler-6d46488bd4-t67r4                       0/1     Pending   0          15h
kube-system       l7-default-backend-7dc957ffb5-8k2gb                        0/1     Pending   0          15h
kube-system       metrics-server-v1.33.0-5d59c97b7b-kttp7                    0/1     Pending   0          15h
kube-system       metrics-server-v1.33.0-77d549c684-w72t9                    0/1     Pending   0          15h


no nodes available
$ kubectl events --all-namespaces
NAMESPACE         LAST SEEN                TYPE      REASON                           OBJECT                                                         MESSAGE
gke-gmp-system    10m (x124 over 15h)      Normal    DNSRecordProvisioningSucceeded   Service/alertmanager                                           DNS records updated
gke-managed-cim   4m23s (x3646 over 15h)   Warning   FailedGetResourceMetric          HorizontalPodAutoscaler/kube-state-metrics                     unable to get metric memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
kube-system       90s (x4939 over 15h)     Warning   FailedScheduling                 Pod/filestore-lock-release-controller-55fb8b59f5-gkt9w         no nodes available to schedule pods
kube-system       89s (x4939 over 15h)     Warning   FailedScheduling                 Pod/event-exporter-gke-78cfd77cbf-rm6fb                        no nodes available to schedule pods
kube-system       89s (x4939 over 15h)     Warning   FailedScheduling                 Pod/kube-dns-594d77f8c4-tcwc9                                  no nodes available to schedule pods
gke-managed-cim   89s (x4939 over 15h)     Warning   FailedScheduling                 Pod/kube-state-metrics-0                                       no nodes available to schedule pods
kube-system       56s (x4939 over 15h)     Warning   FailedScheduling                 Pod/kube-dns-autoscaler-6d46488bd4-t67r4                       no nodes available to schedule pods
kube-system       56s (x4940 over 15h)     Warning   FailedScheduling                 Pod/konnectivity-agent-85988db6dd-szr8m                        no nodes available to schedule pods
kube-system       56s (x4939 over 15h)     Warning   FailedScheduling                 Pod/konnectivity-agent-autoscaler-8567d9c95d-2dskj             no nodes available to schedule pods
kube-system       56s (x4939 over 15h)     Warning   FailedScheduling                 Pod/l7-default-backend-7dc957ffb5-8k2gb                        no nodes available to schedule pods
gke-gmp-system    45s (x4939 over 15h)     Warning   FailedScheduling                 Pod/alertmanager-0                                             no nodes available to schedule pods
gke-gmp-system    45s (x4939 over 15h)     Warning   FailedScheduling                 Pod/gmp-operator-59bb7c6ddb-tm4vs                              no nodes available to schedule pods
gke-gmp-system    45s (x4939 over 15h)     Warning   FailedScheduling                 Pod/rule-evaluator-6c5bc8d6dc-v4b42                            no nodes available to schedule pods
kube-system       34s (x4939 over 15h)     Warning   FailedScheduling                 Pod/egress-nat-controller-86c7bc9c6d-x65gv                     no nodes available to schedule pods
kube-system       34s (x4939 over 15h)     Warning   FailedScheduling                 Pod/metrics-server-v1.33.0-5d59c97b7b-kttp7                    no nodes available to schedule pods
kube-system       34s (x4939 over 15h)     Warning   FailedScheduling                 Pod/metrics-server-v1.33.0-77d549c684-w72t9                    no nodes available to schedule pods
kube-system       23s (x4939 over 15h)     Warning   FailedScheduling                 Pod/antrea-controller-horizontal-autoscaler-54745fb775-sf6pb   no nodes available to schedule pods

Try a 2nd cluster in us-east4 (Virginia) with 10.1.0.0/17 cidr

20251029

$ gcloud beta container --project "biometric-backend-gke-man-old" clusters create-auto "dev" --region "us-east4" --release-channel "stable" --enable-private-nodes --enable-dns-access --enable-k8s-tokens-via-dns --enable-k8s-certs-via-dns --enable-ip-access --enable-master-global-access --enable-google-cloud-access --network "projects/biometric-backend-gke-man-old/global/networks/default" --subnetwork "projects/biometric-backend-gke-man-old/regions/us-east4/subnetworks/default" --cluster-ipv4-cidr "10.1.0.0/17" --binauthz-evaluation-mode=DISABLED --scopes=https://www.googleapis.com/auth/cloud-platform --enable-secret-manager
Note: This cluster has private nodes. If you need connectivity to the public internet, for example to pull public containers, you must configure Cloud NAT. To enable NAT for the network of this cluster, run the following commands:
gcloud compute routers create my-router --region us-east4 --network default --project=biometric-backend-gke-man-old
gcloud beta compute routers nats create nat --router=my-router --region=us-east4 --auto-allocate-nat-external-ips --nat-all-subnet-ip-ranges --project=biometric-backend-gke-man-old
Creating cluster dev in us-east4...
..................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................done.
Created [https://container.googleapis.com/v1beta1/projects/biometric-backend-gke-man-old/zones/us-east4/clusters/dev].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east4/dev?project=biometric-backend-gke-man-old
kubeconfig entry generated for dev.
NAME  LOCATION  MASTER_VERSION      MASTER_IP       MACHINE_TYPE   NODE_VERSION        NUM_NODES  STATUS   STACK_TYPE
dev   us-east4  1.33.4-gke.1350000  34.150.174.245  ek-standard-8  1.33.4-gke.1350000  3          RUNNING  IPV4

We have nodes this time 

$ gcloud container clusters get-credentials dev --region us-east4 --project biometric-backend-gke-man-old --dns-endpoint
Fetching cluster endpoint and auth data.
kubeconfig entry generated for dev.

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get nodes
NAME                                 STATUS     ROLES    AGE     VERSION
gk3-dev-default-pool-d141939e-bzz5   NotReady   <none>   4m12s   v1.33.4-gke.1350000



kube-system                    65s                     Warning   NodeNotReady                     Pod/gcsfusecsi-node-pwzc4                                         Node is not ready
kube-system                    65s                     Warning   NodeNotReady                     Pod/gke-metadata-server-k7qkj                                     Node is not ready
kube-system                    65s                     Warning   NodeNotReady                     Pod/parallelstore-csi-node-w4npw                                  Node is not ready
kube-system                    65s                     Warning   NodeNotReady                     Pod/efficiency-daemon-l6hdw                                       Node is not ready
kube-system                    65s                     Warning   NodeNotReady                     Pod/pdcsi-node-qb9k4                                              Node is not ready
kube-system                    64s                     Warning   NodeNotReady                     Pod/anetd-m-67ss8                                                 Node is not ready
kube-system                    64s                     Warning   NodeNotReady                     Pod/csi-secrets-store-gke-4gbg6                                   Node is not ready
kube-system                    64s                     Warning   NodeNotReady                     Pod/gke-system-balloon-pod-44mh6                                  Node is not ready
kube-system                    64s                     Warning   NodeNotReady                     Pod/filestore-node-g22v2                                          Node is not ready
gke-managed-cim                11s (x10 over 86s)      Warning   FailedGetResourceMetric          HorizontalPodAutoscaler/kube-state-metrics                        unable to get metric memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
default                        0s                      Normal    DeletingNode                     Node/gk3-dev-default-pool-d141939e-bzz5                           Deleting node gk3-dev-default-pool-d141939e-bzz5 because it does not exist in the cloud provider

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get pods --all-namespaces
NAMESPACE         NAME                                                       READY   STATUS     RESTARTS   AGE
gke-gmp-system    alertmanager-0                                             0/2     Pending    0          5m5s
gke-gmp-system    collector-bcsvx                                            0/2     Init:0/2   0          4m40s
gke-gmp-system    gmp-operator-74cfc48984-z85q6                              0/1     Pending    0          5m5s
gke-gmp-system    rule-evaluator-85d74c47d4-cl6n2                            0/2     Pending    0          5m5s
gke-managed-cim   kube-state-metrics-0                                       0/2     Pending    0          6m15s
kube-system       anetd-m-67ss8                                              3/3     Running    0          4m40s
kube-system       antrea-controller-horizontal-autoscaler-7555bc5748-jb4wb   0/1     Pending    0          4m46s
kube-system       csi-secrets-store-gke-4gbg6                                3/3     Running    0          4m25s
kube-system       csi-secrets-store-provider-gke-x5clc                       1/1     Running    0          4m24s
kube-system       efficiency-daemon-l6hdw                                    1/1     Running    0          4m40s
kube-system       egress-nat-controller-6556c78ffc-lcgsv                     0/1     Pending    0          4m53s
kube-system       event-exporter-gke-545dc779f5-77kdb                        0/2     Pending    0          6m7s
kube-system       filestore-lock-release-controller-6cd47fcbc6-8v5sn         0/2     Pending    0          5m57s
kube-system       filestore-node-g22v2                                       4/4     Running    0          4m40s
kube-system       fluentbit-gke-big-x66zx                                    3/3     Running    0          4m40s
kube-system       gcsfusecsi-node-pwzc4                                      3/3     Running    0          4m40s
kube-system       gke-metadata-server-k7qkj                                  1/1     Running    0          4m40s
kube-system       gke-metrics-agent-rrrzz                                    3/3     Running    0          4m40s
kube-system       gke-system-balloon-pod-44mh6                               1/1     Running    0          3m22s
kube-system       ip-masq-agent-t8w2f                                        1/1     Running    0          4m40s
kube-system       konnectivity-agent-autoscaler-779746dccc-wv2dt             0/1     Pending    0          5m19s
kube-system       konnectivity-agent-cd9c7f75b-8vpsm                         0/2     Pending    0          5m21s
kube-system       kube-dns-97b47d4f-qs964                                    0/4     Pending    0          6m19s
kube-system       kube-dns-autoscaler-654f48d7dc-lpvgx                       0/1     Pending    0          5m17s
kube-system       l7-default-backend-6ffffb67b8-xv22z                        0/1     Pending    0          5m14s
kube-system       metrics-server-v1.33.0-544754f487-7mqbt                    0/1     Pending    0          4m55s
kube-system       metrics-server-v1.33.0-86b7756575-97z6b                    0/1     Pending    0          4m57s
kube-system       netd-dhpfh                                                 3/3     Running    0          4m40s
kube-system       node-local-dns-9hlkt                                       2/2     Running    0          4m39s
kube-system       parallelstore-csi-node-w4npw                               2/2     Running    0          4m37s
kube-system       pdcsi-node-qb9k4                                           2/2     Running    0          4m32s


reverted
micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get pods --all-namespaces
NAMESPACE         NAME                                                       READY   STATUS    RESTARTS   AGE
gke-gmp-system    alertmanager-0                                             0/2     Pending   0          7m38s
gke-gmp-system    gmp-operator-74cfc48984-z85q6                              0/1     Pending   0          7m38s
gke-gmp-system    rule-evaluator-85d74c47d4-cl6n2                            0/2     Pending   0          7m38s
gke-managed-cim   kube-state-metrics-0                                       0/2     Pending   0          8m48s
kube-system       antrea-controller-horizontal-autoscaler-7555bc5748-jb4wb   0/1     Pending   0          7m19s
kube-system       egress-nat-controller-6556c78ffc-lcgsv                     0/1     Pending   0          7m26s
kube-system       event-exporter-gke-545dc779f5-77kdb                        0/2     Pending   0          8m40s
kube-system       filestore-lock-release-controller-6cd47fcbc6-8v5sn         0/2     Pending   0          8m30s
kube-system       konnectivity-agent-autoscaler-779746dccc-wv2dt             0/1     Pending   0          7m52s
kube-system       konnectivity-agent-cd9c7f75b-8vpsm                         0/2     Pending   0          7m54s
kube-system       kube-dns-97b47d4f-qs964                                    0/4     Pending   0          8m52s
kube-system       kube-dns-autoscaler-654f48d7dc-lpvgx                       0/1     Pending   0          7m50s
kube-system       l7-default-backend-6ffffb67b8-xv22z                        0/1     Pending   0          7m47s
kube-system       metrics-server-v1.33.0-544754f487-7mqbt                    0/1     Pending   0          7m28s
kube-system       metrics-server-v1.33.0-86b7756575-97z6b                    0/1     Pending   0          7m30s

gke-gmp-system                 8m29s (x5 over 10m)      Warning   FailedMount                      Pod/collector-bcsvx                                               MountVolume.SetUp failed for volume "collection-secret" : secret "collection" not found
gke-gmp-system                 8m29s (x5 over 10m)      Warning   FailedMount                      Pod/collector-bcsvx                                               MountVolume.SetUp failed for volume "config" : configmap "collector" not found
gke-gmp-system                 8m20s                    Normal    ADD                              Service/rule-evaluator                                            gke-gmp-system/rule-evaluator
kube-system                    8m20s                    Normal    ADD                              Service/default-http-backend                                      kube-system/default-http-backend
kube-system                    8m20s                    Normal    ADD                              Service/metrics-server                                            kube-system/metrics-server
kube-system                    8m20s                    Normal    ADD                              Service/kube-dns                                                  kube-system/kube-dns
default                        8m20s                    Normal    ADD                              GatewayClass/gke-l7-gxlb                                          gke-l7-gxlb
default                        8m20s                    Normal    ADD                              GatewayClass/gke-l7-rilb                                          gke-l7-rilb
kube-system                    8m20s                    Normal    ADD                              Service/antrea                                                    kube-system/antrea
gke-gmp-system                 8m20s                    Normal    ADD                              Service/alertmanager                                              gke-gmp-system/alertmanager
default                        8m20s                    Normal    ADD                              GatewayClass/gke-l7-global-external-managed                       gke-l7-global-external-managed
gke-gmp-system                 8m20s                    Normal    ADD                              Service/gmp-operator                                              gke-gmp-system/gmp-operator
default                        8m20s                    Normal    ADD                              GatewayClass/gke-l7-regional-external-managed                     gke-l7-regional-external-managed
default                        8m20s                    Normal    ADD                              Service/kubernetes                                                default/kubernetes
default                        8m3s                     Normal    ScaleDown                        Node/gk3-dev-default-pool-d141939e-bzz5                           marked the node as toBeDeleted/unschedulable
kube-system                    8m3s                     Normal    ScaleDownEmpty                   ConfigMap/cluster-autoscaler-status                               Scale-down: removing empty node "gk3-dev-default-pool-d141939e-bzz5"
kube-system                    8m2s                     Normal    ScaleDownEmpty                   ConfigMap/cluster-autoscaler-status                               Scale-down: empty node gk3-dev-default-pool-d141939e-bzz5 removed
gke-managed-cim                7m50s (x8 over 9m24s)    Warning   FailedGetResourceMetric          HorizontalPodAutoscaler/kube-state-metrics                        did not receive metrics for targeted pods (pods might be unready)
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/efficiency-daemon-l6hdw                                       Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/gcsfusecsi-node-pwzc4                                         Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/gke-metrics-agent-rrrzz                                       Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/netd-dhpfh                                                    Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/node-local-dns-9hlkt                                          Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/fluentbit-gke-big-x66zx                                       Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/ip-masq-agent-t8w2f                                           Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/pdcsi-node-qb9k4                                              Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/gke-metadata-server-k7qkj                                     Node is not ready
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/csi-secrets-store-provider-gke-x5clc                          Node is not ready
default                        7m14s                    Normal    NodeNotReady                     Node/gk3-dev-default-pool-d141939e-bzz5                           Node gk3-dev-default-pool-d141939e-bzz5 status is now: NodeNotReady
kube-system                    7m14s                    Warning   NodeNotReady                     Pod/parallelstore-csi-node-w4npw                                  Node is not ready
kube-system                    7m13s                    Warning   NodeNotReady                     Pod/anetd-m-67ss8                                                 Node is not ready
kube-system                    7m13s                    Warning   NodeNotReady                     Pod/gke-system-balloon-pod-44mh6                                  Node is not ready
kube-system                    7m13s                    Warning   NodeNotReady                     Pod/csi-secrets-store-gke-4gbg6                                   Node is not ready
kube-system                    7m13s                    Warning   NodeNotReady                     Pod/filestore-node-g22v2                                          Node is not ready
default                        6m9s                     Normal    DeletingNode                     Node/gk3-dev-default-pool-d141939e-bzz5                           Deleting node gk3-dev-default-pool-d141939e-bzz5 because it does not exist in the cloud provider
default                        6m8s                     Normal    RemovingNode                     Node/gk3-dev-default-pool-d141939e-bzz5                           Node gk3-dev-default-pool-d141939e-bzz5 event: Removing Node gk3-dev-default-pool-d141939e-bzz5 from Controller
gke-managed-cim                4m35s (x18 over 7m35s)   Warning   FailedGetResourceMetric          HorizontalPodAutoscaler/kube-state-metrics                        unable to get metric memory: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)
kube-system                    102s (x30 over 11m)      Warning   FailedScheduling                 Pod/filestore-lock-release-controller-6cd47fcbc6-8v5sn            no nodes available to schedule pods
kube-system                    102s (x30 over 11m)      Warning   FailedScheduling                 Pod/event-exporter-gke-545dc779f5-77kdb                           no nodes available to schedule pods
gke-managed-cim                102s (x30 over 11m)      Warning   FailedScheduling                 Pod/kube-state-metrics-0                                          no nodes available to schedule pods
kube-system                    102s (x30 over 11m)      Warning   FailedScheduling                 Pod/kube-dns-97b47d4f-qs964                                       no nodes available to schedule pods
kube-system                    69s (x30 over 11m)       Warning   FailedScheduling                 Pod/l7-default-backend-6ffffb67b8-xv22z                           no nodes available to schedule pods
kube-system                    69s (x30 over 11m)       Warning   FailedScheduling                 Pod/konnectivity-agent-cd9c7f75b-8vpsm                            no nodes available to schedule pods
kube-system                    69s (x30 over 11m)       Warning   FailedScheduling                 Pod/konnectivity-agent-autoscaler-779746dccc-wv2dt                no nodes available to schedule pods
kube-system                    69s (x30 over 11m)       Warning   FailedScheduling                 Pod/kube-dns-autoscaler-654f48d7dc-lpvgx                          no nodes available to schedule pods
gke-gmp-system                 58s (x30 over 11m)       Warning   FailedScheduling                 Pod/alertmanager-0                                                no nodes available to schedule pods
gke-gmp-system                 58s (x30 over 11m)       Warning   FailedScheduling                 Pod/rule-evaluator-85d74c47d4-cl6n2                               no nodes available to schedule pods
gke-gmp-system                 58s (x30 over 11m)       Warning   FailedScheduling                 Pod/gmp-operator-74cfc48984-z85q6                                 no nodes available to schedule pods
kube-system                    47s (x30 over 10m)       Warning   FailedScheduling                 Pod/metrics-server-v1.33.0-86b7756575-97z6b                       no nodes available to schedule pods
kube-system                    47s (x30 over 10m)       Warning   FailedScheduling                 Pod/metrics-server-v1.33.0-544754f487-7mqbt                       no nodes available to schedule pods
kube-system                    47s (x30 over 10m)       Warning   FailedScheduling                 Pod/egress-nat-controller-6556c78ffc-lcgsv                        no nodes available to schedule pods
kube-system                    36s (x30 over 10m)       Warning   FailedScheduling                 Pod/antrea-controller-horizontal-autoscaler-7555bc5748-jb4wb      no nodes available to schedule pods


1153
micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ kubectl get events
LAST SEEN   TYPE      REASON                           OBJECT                                          MESSAGE
29m         Warning   InvalidDiskCapacity              node/gk3-dev-default-pool-d141939e-bzz5         invalid capacity 0 on image filesystem
29m         Normal    NodeHasSufficientMemory          node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 status is now: NodeHasSufficientMemory
29m         Normal    NodeHasNoDiskPressure            node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 status is now: NodeHasNoDiskPressure
29m         Normal    NodeHasSufficientPID             node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 status is now: NodeHasSufficientPID
29m         Normal    NodeAllocatableEnforced          node/gk3-dev-default-pool-d141939e-bzz5         Updated Node Allocatable limit across pods
29m         Warning   NodeRegistrationCheckerStart     node/gk3-dev-default-pool-d141939e-bzz5         Wed Oct 29 15:22:34 UTC 2025 - ** Starting Node Registration Checker **
29m         Warning   ContainerdStart                  node/gk3-dev-default-pool-d141939e-bzz5         Starting containerd.service...
29m         Warning   DockerStart                      node/gk3-dev-default-pool-d141939e-bzz5         Starting docker.service...
27m         Warning   EKBalloonPodIsNotRunning         node/gk3-dev-default-pool-d141939e-bzz5
29m         Warning   KubeletStart                     node/gk3-dev-default-pool-d141939e-bzz5         Started kubelet.service.
29m         Normal    EKKubeProxyPodNotDetected        node/gk3-dev-default-pool-d141939e-bzz5         Node condition EKKubeProxyNotSupported is now: Unknown, reason: EKKubeProxyPodNotDetected, message: "Error checking for kube-proxy pod: [Errno 2] No such file or directory"
26m         Normal    RegisteredNode                   node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 event: Registered Node gk3-dev-default-pool-d141939e-bzz5 in Controller
26m         Normal    Synced                           node/gk3-dev-default-pool-d141939e-bzz5         Node synced successfully
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830 is backed by image streaming.
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830 is backed by secondary boot disk caching by 0.0% (0/2 layers), by image streaming by 100.0% (2/2 layers).
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/node_token_broker/init@sha256:37e96e6fdb960fd60b14599e4aca02f4c4e46dac9700f27500e06f9442ab1046 is backed by image streaming.
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/node_token_broker/init@sha256:37e96e6fdb960fd60b14599e4aca02f4c4e46dac9700f27500e06f9442ab1046 is backed by secondary boot disk caching by 0.0% (0/1 layers), by image streaming by 100.0% (1/1 layers).
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Layer download failed. Layer id: "sha256:d7dfcfd89e8a42f4fed57a8d9cc9077cdf11c83f60d2407f570b535eb8efc79c", image: "us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830"
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/csi-node-driver-registrar@sha256:7394b701866d6f10296dd1652ca6f29ab690ea528d75327076d8a4ca5a9f5ca7 is backed by image streaming.
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/csi-node-driver-registrar@sha256:7394b701866d6f10296dd1652ca6f29ab690ea528d75327076d8a4ca5a9f5ca7 is backed by secondary boot disk caching by 0.0% (0/8 layers), by image streaming by 62.5% (5/8 layers).
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-node-cache@sha256:b986fcce51eb582dfa1b231034c1c391107aa06e2cd4bab4cae49c91db463074 is backed by image streaming.
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-node-cache@sha256:b986fcce51eb582dfa1b231034c1c391107aa06e2cd4bab4cae49c91db463074 is backed by secondary boot disk caching by 0.0% (0/6 layers), by image streaming by 100.0% (6/6 layers).
26m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Layer download failed. Layer id: "sha256:780b0a606b52e6eaa01eb417e73bf0aa427ecbca40e44a5337d5a817bbe96e8c", image: "us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830"
25m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/efficiency-daemon@sha256:f24853e48821137fbccac9946c1740bf51c66ba3ff09e2a5f5e35b1aab1ebcbf is backed by image streaming.
25m         Normal    ImageStreaming                   node/gk3-dev-default-pool-d141939e-bzz5         (combined from similar events): Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/gcs-fuse-csi-driver@sha256:bcef473c72114d89e3aa45c017314ed60d2f4bcf9a8257fb979cf236310bafa8 is backed by secondary boot disk caching by 0.0% (0/23 layers), by image streaming by 82.6% (19/23 layers).
24m         Normal    EKKubeProxyPodNotDetected        node/gk3-dev-default-pool-d141939e-bzz5         Node condition EKKubeProxyNotSupported is now: False, reason: EKKubeProxyPodNotDetected, message: "unsupported kube-proxy pod in EK family not detected"
23m         Normal    ScaleDown                        node/gk3-dev-default-pool-d141939e-bzz5         marked the node as toBeDeleted/unschedulable
22m         Normal    NodeNotReady                     node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 status is now: NodeNotReady
21m         Normal    DeletingNode                     node/gk3-dev-default-pool-d141939e-bzz5         Deleting node gk3-dev-default-pool-d141939e-bzz5 because it does not exist in the cloud provider
21m         Normal    RemovingNode                     node/gk3-dev-default-pool-d141939e-bzz5         Node gk3-dev-default-pool-d141939e-bzz5 event: Removing Node gk3-dev-default-pool-d141939e-bzz5 from Controller
24m         Normal    ADD                              gatewayclass/gke-l7-global-external-managed     gke-l7-global-external-managed
23m         Normal    ADD                              gatewayclass/gke-l7-global-external-managed     gke-l7-global-external-managed
24m         Normal    ADD                              gatewayclass/gke-l7-gxlb                        gke-l7-gxlb
23m         Normal    ADD                              gatewayclass/gke-l7-gxlb                        gke-l7-gxlb
23m         Normal    ADD                              gatewayclass/gke-l7-regional-external-managed   gke-l7-regional-external-managed
24m         Normal    ADD                              gatewayclass/gke-l7-rilb                        gke-l7-rilb
23m         Normal    ADD                              gatewayclass/gke-l7-rilb                        gke-l7-rilb
27m         Normal    DNSRecordProvisioningSucceeded   service/kubernetes                              DNS records updated
24m         Normal    ADD                              service/kubernetes                              default/kubernetes
23m         Normal    ADD                              service/kubernetes                              default/kubernetes


It may be that scaling will occur on first pod deploy

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ kubectl get pods --all-namespaces
NAMESPACE         NAME                                                       READY   STATUS    RESTARTS   AGE
gke-gmp-system    alertmanager-0                                             0/2     Pending   0          29m
gke-gmp-system    gmp-operator-6b88894688-ggvmf                              0/1     Pending   0          113s
gke-gmp-system    gmp-operator-74cfc48984-z85q6                              0/1     Pending   0          29m
gke-gmp-system    rule-evaluator-85d74c47d4-cl6n2                            0/2     Pending   0          29m
gke-managed-cim   kube-state-metrics-0                                       0/2     Pending   0          30m
kube-system       antrea-controller-horizontal-autoscaler-7555bc5748-jb4wb   0/1     Pending   0          28m
kube-system       egress-nat-controller-6556c78ffc-lcgsv                     0/1     Pending   0          29m
kube-system       event-exporter-gke-545dc779f5-77kdb                        0/2     Pending   0          30m
kube-system       filestore-lock-release-controller-6cd47fcbc6-8v5sn         0/2     Pending   0          30m
kube-system       konnectivity-agent-autoscaler-779746dccc-wv2dt             0/1     Pending   0          29m
kube-system       konnectivity-agent-cd9c7f75b-8vpsm                         0/2     Pending   0          29m
kube-system       kube-dns-97b47d4f-qs964                                    0/4     Pending   0          30m
kube-system       kube-dns-autoscaler-654f48d7dc-lpvgx                       0/1     Pending   0          29m
kube-system       l7-default-backend-6ffffb67b8-xv22z                        0/1     Pending   0          29m
kube-system       metrics-server-v1.33.0-544754f487-7mqbt                    0/1     Pending   0          29m
kube-system       metrics-server-v1.33.0-86b7756575-97z6b                    0/1     Pending   0          29m

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ cd kubernetes-gke/

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl apply -f namespace.yaml
namespace/magellan created

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get namespaces
NAME                           STATUS   AGE
default                        Active   35m
gke-gmp-system                 Active   31m
gke-managed-cim                Active   32m
gke-managed-filestorecsi       Active   32m
gke-managed-parallelstorecsi   Active   30m
gke-managed-system             Active   31m
gke-managed-volumepopulator    Active   31m
gmp-public                     Active   31m
kube-node-lease                Active   35m
kube-public                    Active   35m
kube-system                    Active   35m
magellan                       Active   9s

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl apply -f deployment.yaml
Warning: autopilot-default-resources-mutator:Autopilot updated Deployment magellan/magellan-nbi: defaulted unspecified 'cpu' resource for containers [biometric-nbi] (see http://g.co/gke/autopilot-defaults).
deployment.apps/magellan-nbi created

wait an hour
$ kubectl get pods -n magellan
NAME                          READY   STATUS    RESTARTS   AGE
magellan-nbi-d9f5cb64-p4zr8   1/1     Running   0          89m

$ kubectl get events
LAST SEEN   TYPE      REASON                                   OBJECT                              MESSAGE
12m         Normal    ScaleDown                                node/gk3-dev-pool-2-4b0da102-cr2d   marked the node as toBeDeleted/unschedulable
11m         Warning   EKBalloonPodIsNotRunning                 node/gk3-dev-pool-2-4b0da102-cr2d   gke-system-balloon-pod is not running for more than 20 seconds
9m19s       Normal    NodeNotReady                             node/gk3-dev-pool-2-4b0da102-cr2d   Node gk3-dev-pool-2-4b0da102-cr2d status is now: NodeNotReady
9m15s       Normal    DeletingNode                             node/gk3-dev-pool-2-4b0da102-cr2d   Deleting node gk3-dev-pool-2-4b0da102-cr2d because it does not exist in the cloud provider
9m13s       Normal    RemovingNode                             node/gk3-dev-pool-2-4b0da102-cr2d   Node gk3-dev-pool-2-4b0da102-cr2d event: Removing Node gk3-dev-pool-2-4b0da102-cr2d from Controller
12m         Normal    Starting                                 node/gk3-dev-pool-2-a2dfdf6f-zmbf   Starting kubelet.
12m         Warning   InvalidDiskCapacity                      node/gk3-dev-pool-2-a2dfdf6f-zmbf   invalid capacity 0 on image filesystem
12m         Normal    NodeHasSufficientMemory                  node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node gk3-dev-pool-2-a2dfdf6f-zmbf status is now: NodeHasSufficientMemory
12m         Normal    NodeHasNoDiskPressure                    node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node gk3-dev-pool-2-a2dfdf6f-zmbf status is now: NodeHasNoDiskPressure
12m         Normal    NodeHasSufficientPID                     node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node gk3-dev-pool-2-a2dfdf6f-zmbf status is now: NodeHasSufficientPID
12m         Normal    NodeAllocatableEnforced                  node/gk3-dev-pool-2-a2dfdf6f-zmbf   Updated Node Allocatable limit across pods
12m         Warning   NodeRegistrationCheckerStart             node/gk3-dev-pool-2-a2dfdf6f-zmbf   Wed Oct 29 17:14:25 UTC 2025 - ** Starting Node Registration Checker **
12m         Warning   ContainerdStart                          node/gk3-dev-pool-2-a2dfdf6f-zmbf   Starting containerd.service...
12m         Warning   DockerStart                              node/gk3-dev-pool-2-a2dfdf6f-zmbf   Starting docker.service...
12m         Warning   KubeletStart                             node/gk3-dev-pool-2-a2dfdf6f-zmbf   Started kubelet.service.
12m         Normal    Synced                                   node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node synced successfully
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830 is backed by image streaming.
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830 is backed by secondary boot disk caching by 0.0% (0/2 layers), by image streaming by 100.0% (2/2 layers).
12m         Normal    RegisteredNode                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node gk3-dev-pool-2-a2dfdf6f-zmbf event: Registered Node gk3-dev-pool-2-a2dfdf6f-zmbf in Controller
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/node_token_broker/init@sha256:37e96e6fdb960fd60b14599e4aca02f4c4e46dac9700f27500e06f9442ab1046 is backed by image streaming.
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/node_token_broker/init@sha256:37e96e6fdb960fd60b14599e4aca02f4c4e46dac9700f27500e06f9442ab1046 is backed by secondary boot disk caching by 0.0% (0/1 layers), by image streaming by 100.0% (1/1 layers).
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/csi-node-driver-registrar@sha256:7394b701866d6f10296dd1652ca6f29ab690ea528d75327076d8a4ca5a9f5ca7 is backed by image streaming.
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/csi-node-driver-registrar@sha256:7394b701866d6f10296dd1652ca6f29ab690ea528d75327076d8a4ca5a9f5ca7 is backed by secondary boot disk caching by 0.0% (0/8 layers), by image streaming by 62.5% (5/8 layers).
12m         Normal    NodeReady                                node/gk3-dev-pool-2-a2dfdf6f-zmbf   Node gk3-dev-pool-2-a2dfdf6f-zmbf status is now: NodeReady
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/gke-metadata-server@sha256:b136c75032e99c2923ff088abeed425a0098a00f0a73f66afce7f3267d00652f is not backed by Image Streaming.
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Layer download failed. Layer id: "sha256:780b0a606b52e6eaa01eb417e73bf0aa427ecbca40e44a5337d5a817bbe96e8c", image: "us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830"
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Layer download failed. Layer id: "sha256:d7dfcfd89e8a42f4fed57a8d9cc9077cdf11c83f60d2407f570b535eb8efc79c", image: "us-east4-artifactregistry.gcr.io/gke-release/gke-release/pause@sha256:880e63f94b145e46f1b1082bb71b85e21f16b99b180b9996407d61240ceb9830"
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/efficiency-daemon@sha256:f24853e48821137fbccac9946c1740bf51c66ba3ff09e2a5f5e35b1aab1ebcbf is backed by image streaming.
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/efficiency-daemon@sha256:f24853e48821137fbccac9946c1740bf51c66ba3ff09e2a5f5e35b1aab1ebcbf is backed by secondary boot disk caching by 0.0% (0/16 layers), by image streaming by 25.0% (4/16 layers).
12m         Normal    ImageStreaming                           node/gk3-dev-pool-2-a2dfdf6f-zmbf   (combined from similar events): Image us-east4-artifactregistry.gcr.io/gke-release/gke-release/proxy-agent@sha256:aa89e06b001dd7ef9e9082de3da7ef4b7995c6c857e905d3810eb9901664158f is backed by secondary boot disk caching by 0.0% (0/15 layers), by image streaming by 20.0% (3/15 layers).
5m32s       Warning   NodeRegistrationCheckerDidNotRunChecks   node/gk3-dev-pool-2-a2dfdf6f-zmbf   Wed Oct 29 17:21:25 UTC 2025 - **     Node ready and registered. **


micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get pods --all-namespaces
NAMESPACE         NAME                                                       READY   STATUS    RESTARTS   AGE
gke-gmp-system    collector-6nc8w                                            2/2     Running   0          14m
gke-gmp-system    collector-q4w7w                                            2/2     Running   0          90m
gke-gmp-system    gmp-operator-6b88894688-ggvmf                              1/1     Running   0          95m
gke-managed-cim   kube-state-metrics-0                                       2/2     Running   0          124m
kube-system       anetd-l-4tx8r                                              3/3     Running   0          14m
kube-system       anetd-l-8ngbv                                              3/3     Running   0          90m
kube-system       antrea-controller-horizontal-autoscaler-7555bc5748-jb4wb   1/1     Running   0          122m
kube-system       csi-secrets-store-gke-28jvm                                3/3     Running   0          14m
kube-system       csi-secrets-store-gke-tlks7                                3/3     Running   0          90m
kube-system       csi-secrets-store-provider-gke-cslsd                       1/1     Running   0          90m
kube-system       csi-secrets-store-provider-gke-r9vbp                       1/1     Running   0          14m
kube-system       efficiency-daemon-n6sqr                                    1/1     Running   0          90m
kube-system       efficiency-daemon-nhxzq                                    1/1     Running   0          14m
kube-system       egress-nat-controller-6556c78ffc-lcgsv                     1/1     Running   0          122m
kube-system       event-exporter-gke-545dc779f5-77kdb                        2/2     Running   0          124m
kube-system       filestore-lock-release-controller-6cd47fcbc6-8v5sn         2/2     Running   0          123m
kube-system       filestore-node-7xzqs                                       4/4     Running   0          90m
kube-system       filestore-node-nwc8t                                       4/4     Running   0          14m
kube-system       fluentbit-gke-big-5sh4h                                    3/3     Running   0          90m
kube-system       fluentbit-gke-big-b5hbc                                    3/3     Running   0          14m
kube-system       gcsfusecsi-node-ff7rs                                      3/3     Running   0          90m
kube-system       gcsfusecsi-node-jgh98                                      3/3     Running   0          14m
kube-system       gke-metadata-server-8kjfv                                  1/1     Running   0          90m
kube-system       gke-metadata-server-rcrgq                                  1/1     Running   0          14m
kube-system       gke-metrics-agent-42qpb                                    3/3     Running   0          90m
kube-system       gke-metrics-agent-vvz5b                                    3/3     Running   0          14m
kube-system       gke-system-balloon-pod-2lrn2                               1/1     Running   0          77m
kube-system       gke-system-balloon-pod-d7r6r                               1/1     Running   0          8m14s
kube-system       ip-masq-agent-dmjxn                                        1/1     Running   0          14m
kube-system       ip-masq-agent-jrtkc                                        1/1     Running   0          90m
kube-system       konnectivity-agent-autoscaler-779746dccc-wv2dt             1/1     Running   0          123m
kube-system       konnectivity-agent-cd9c7f75b-8vpsm                         2/2     Running   0          123m
kube-system       konnectivity-agent-cd9c7f75b-jknmc                         2/2     Running   0          13m
kube-system       kube-dns-97b47d4f-m5sql                                    4/4     Running   0          13m
kube-system       kube-dns-97b47d4f-qs964                                    4/4     Running   0          124m
kube-system       kube-dns-autoscaler-654f48d7dc-lpvgx                       1/1     Running   0          123m
kube-system       l7-default-backend-6ffffb67b8-xv22z                        1/1     Running   0          123m
kube-system       metrics-server-v1.33.0-544754f487-7mqbt                    1/1     Running   0          122m
kube-system       netd-fhdk2                                                 3/3     Running   0          14m
kube-system       netd-gv9j9                                                 3/3     Running   0          90m
kube-system       node-local-dns-pqn99                                       2/2     Running   0          14m
kube-system       node-local-dns-xz88k                                       2/2     Running   0          90m
kube-system       parallelstore-csi-node-7xgdr                               2/2     Running   0          90m
kube-system       parallelstore-csi-node-8ffzf                               2/2     Running   0          14m
kube-system       pdcsi-node-24tkm                                           2/2     Running   0          90m
kube-system       pdcsi-node-mmfns                                           2/2     Running   0          14m
magellan          magellan-nbi-d9f5cb64-p4zr8                                1/1     Running   0          91m

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get services --all-namespaces
NAMESPACE        NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
default          kubernetes             ClusterIP   34.118.224.1     <none>        443/TCP            127m
gke-gmp-system   alertmanager           ClusterIP   None             <none>        9093/TCP           123m
gke-gmp-system   gmp-operator           ClusterIP   34.118.239.35    <none>        8443/TCP,443/TCP   123m
gke-gmp-system   rule-evaluator         ClusterIP   34.118.232.173   <none>        19092/TCP          123m
kube-system      antrea                 ClusterIP   34.118.236.110   <none>        443/TCP            123m
kube-system      default-http-backend   NodePort    34.118.236.199   <none>        80:32366/TCP       123m
kube-system      kube-dns               ClusterIP   34.118.224.10    <none>        53/UDP,53/TCP      125m
kube-system      metrics-server         ClusterIP   34.118.239.175   <none>        443/TCP            123m

$ kubectl get nodes
NAME                                 STATUS   ROLES    AGE   VERSION
gk3-dev-nap-dalalgaa-e2cb1349-7vlc   Ready    <none>   91m   v1.33.4-gke.1350000
gk3-dev-pool-2-a2dfdf6f-zmbf         Ready    <none>   15m   v1.33.4-gke.1350000

$ kubectl top nodes
NAME                                 CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
gk3-dev-nap-dalalgaa-e2cb1349-7vlc   238m         1%       3125Mi          5%
gk3-dev-pool-2-a2dfdf6f-zmbf         182m         1%       1688Mi          2%

verify docker image GKE run - private only

add nodeport service, before switching to load balancer

$ kubectl apply -f kubernetes-gke/service.yaml
service/magellan-nbi created

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress (main)
$ kubectl get services -n magellan
NAME           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
magellan-nbi   NodePort   34.118.237.125   <none>        8080:30888/TCP   17s

check exec via cloud shell (not git-bash)

michael@cloudshell:~ (gcp-infrastructure-as-code)$ gcloud container clusters get-credentials dev --region us-east4 --project biometric-backend-gke-man-old --dns-endpoint
Fetching cluster endpoint and auth data.
kubeconfig entry generated for dev.
michael@cloudshell:~ (gcp-infrastructure-as-code)$ kubectl config get-contexts
CURRENT   NAME                                             CLUSTER                                          AUTHINFO                                         NAMESPACE
*         gke_biometric-backend-gke-man-old_us-east4_dev   gke_biometric-backend-gke-man-old_us-east4_dev   gke_biometric-backend-gke-man-old_us-east4_dev 

michael@cloudshell:~ (gcp-infrastructure-as-code)$ kubectl exec -it magellan-nbi-d9f5cb64-p4zr8 -n magellan -- /bin/bash
root@magellan-nbi-d9f5cb64-p4zr8:/# curl http://localhost:8080/nbi/api/health
{"timestamp":"2025-10-29T17:50:44.776+0000","status":404,"error":"Not Found","message":"No message available","path":"/nbi/api/health"}root@magellan-nbi-d9f5cb64-p4zr8:/# curl http://localhost:8080/nbi/app/health
OKroot@magellan-nbi-d9f5cb64-p4zr8:/#

verify docker desktop kubernetes - using dockerhub

$ kubectl config get-contexts
CURRENT   NAME                                             CLUSTER                                          AUTHINFO                                         NAMESPACE
          docker-desktop                                   docker-desktop                                   docker-desktop
          gke_biometric-backend-gke-man-old_us-east1_dev   gke_biometric-backend-gke-man-old_us-east1_dev   gke_biometric-backend-gke-man-old_us-east1_dev
*         gke_biometric-backend-gke-man-old_us-east4_dev   gke_biometric-backend-gke-man-old_us-east4_dev   gke_biometric-backend-gke-man-old_us-east4_dev

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl config use-context docker-desktop
Switched to context "docker-desktop".

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get nodes
NAME             STATUS   ROLES           AGE   VERSION
docker-desktop   Ready    control-plane   17m   v1.34.1

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl apply -f namespace.yaml
namespace/magellan created

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   19m
kube-node-lease   Active   19m
kube-public       Active   19m
kube-system       Active   19m
magellan          Active   7s

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl apply -f deployment.yaml
deployment.apps/magellan-nbi created

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get pods -n magellan
NAME                           READY   STATUS    RESTARTS   AGE
magellan-nbi-c64f988f9-5x62f   1/1     Running   0          14s

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl apply -f service.yaml
service/magellan-nbi created

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-docker (main)
$ kubectl get services -n magellan
NAME           TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
magellan-nbi   NodePort   10.105.125.149   <none>        8080:30888/TCP   8s

http://localhost:30888/nbi/app/health

[obriensystems]

Create artifact registry repository

follow
https://cloud.google.com/artifact-registry/docs/docker/authentication?hl=en&_gl=1*7ibe73*_ga*NzA3MTY2MjI4LjE2OTg2ODk2MTA.*_ga_WH2QY8WWF5*czE3NjE3NDM5MTkkbzY1JGcxJHQxNzYxNzQ1MzAxJGo2MCRsMCRoMA..#before_you_begin

follow
https://cloud.google.com/artifact-registry/docs/docker/authentication?hl=en&_gl=1*7ibe73*_ga*NzA3MTY2MjI4LjE2OTg2ODk2MTA.*_ga_WH2QY8WWF5*czE3NjE3NDM5MTkkbzY1JGcxJHQxNzYxNzQ1MzAxJGo2MCRsMCRoMA..#gcloud-helper

create service account

[obriensystems]

verify docker image GKE run with ingres

references

• https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress

check for httpLoadBalancing plugin - managed by default for autopilot

$ gcloud container clusters describe dev --location=us-east4 --format="value(addonsConfig.httpLoadBalancing.enabled)"

don't use a static IP for now - it will get recreated when the forwarding rule is created
https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress#static_ip_address

https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress#multiple_backend_services

delete the existing nodeport service first

$ kubectl delete -f service.yaml
service "magellan-nbi" deleted from magellan namespace

apply changes in

https://cloud.google.com/kubernetes-engine/docs/how-to/load-balance-ingress#create-ingress

reapply manifests

  kubectl apply -f deployment.yaml
  kubectl apply -f service.yaml

check in ingress.yaml
c2bb10c

$ kubectl apply -f ingress.yaml
Warning: annotation "kubernetes.io/ingress.class" is deprecated, please use 'spec.ingressClassName' instead
ingress.networking.k8s.io/magellan created

missing namespace - defaulting to default

$ kubectl get ingress -n magellan
No resources found in magellan namespace.

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get ingress --all-namespaces
NAMESPACE   NAME       CLASS    HOSTS   ADDRESS   PORTS   AGE
default     magellan   <none>   *                 80      115s

$ kubectl get ingress magellan --output yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce"},"name":"magellan","namespace":"default"},"spec":{"rules":[{"http":{"paths":[{"backend":{"service":{"name":"magellan-nbi","port":{"number":80}}},"path":"/*","pathType":"ImplementationSpecific"}]}}]}}
    kubernetes.io/ingress.class: gce
  creationTimestamp: "2025-10-29T19:19:56Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  name: magellan
  namespace: default
  resourceVersion: "1761765596500719007"
  uid: a0be8e5c-daf6-493b-a653-948047fecfb9
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: magellan-nbi
            port:
              number: 80
        path: /*
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

delete and recreate ingress

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl delete -f ingress.yaml
ingress.networking.k8s.io "magellan" deleted from default namespace

$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/magellan-nbi created

$ kubectl get ingress magellan-nbi -n magellan --output yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"spec.ingressClassName":"gce"},"name":"magellan-nbi","namespace":"magellan"},"spec":{"rules":[{"http":{"paths":[{"backend":{"service":{"name":"magellan-nbi","port":{"number":80}}},"path":"/*","pathType":"ImplementationSpecific"}]}}]}}
    spec.ingressClassName: gce
  creationTimestamp: "2025-10-29T19:26:29Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  name: magellan-nbi
  namespace: magellan
  resourceVersion: "1761765989448255011"
  uid: e720fa70-c260-4a03-b7a4-c16503fb18f9
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: magellan-nbi
            port:
              number: 80
        path: /*
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get ingress -n magellan
NAME           CLASS    HOSTS   ADDRESS   PORTS   AGE
magellan-nbi   <none>   *                 80      2m50s

1528 creating load balancer

1534

Error syncing to GCP: error running backend syncing routine: googleapi: Error 404: The resource 'projects/biometric-backend-gke-man-old/zones/us-east4-a/networkEndpointGroups/k8s1-a8332252-kube-system-default-http-backend-80-199b84eb' was not found, notFound

k8s1-a8332252-kube-system-default-http-backend-80-199b84eb | Backend service (Classic) | Global | HTTP |   |   |   -- | -- | -- | -- | -- | -- | --   | k8s1-a8332252-magellan-magellan-nbi-80-a3759aec | Backend service (Classic) | Global | HTTP

k8s1-a8332252-kube-system-default-http-backend-80-199b84eb Backend service (Classic) Global HTTP
k8s1-a8332252-magellan-magellan-nbi-80-a3759aec Backend service (Classic) Global HTTP

reapply switch Port to 8080 from 80

21m         Normal    ScaleDown                                node/gk3-dev-pool-2-a2dfdf6f-zmbf         marked the node as toBeDeleted/unschedulable
20m         Warning   EKBalloonPodIsNotRunning                 node/gk3-dev-pool-2-a2dfdf6f-zmbf         gke-system-balloon-pod is not running for more than 20 seconds
18m         Normal    NodeNotReady                             node/gk3-dev-pool-2-a2dfdf6f-zmbf         Node gk3-dev-pool-2-a2dfdf6f-zmbf status is now: NodeNotReady
17m         Normal    DeletingNode                             node/gk3-dev-pool-2-a2dfdf6f-zmbf         Deleting node gk3-dev-pool-2-a2dfdf6f-zmbf because it does not exist in the cloud provider
17m         Normal    RemovingNode                             node/gk3-dev-pool-2-a2dfdf6f-zmbf         Node gk3-dev-pool-2-a2dfdf6f-zmbf event: Removing Node gk3-dev-pool-2-a2dfdf6f-zmbf from Controller
16m         Normal    Sync                                     ingress/magellan                          Scheduled for sync
18m         Warning   Translate                                ingress/magellan                          Translation failed: invalid ingress spec: could not find service "default/magellan-nbi"


think I am missing the namespace magellan/magellan-nbi - above (see default) - delete / recreate ingress

        backend:
          service:
            name: magellan-nbi
            namespace: magellan


$ kubectl apply -f deployment.yaml
deployment.apps/magellan-nbi configured

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl apply -f service.yaml
service/magellan-nbi configured

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/magellan-nbi configured


nope
$ kubectl apply -f ingress.yaml
Error from server (BadRequest): error when creating "ingress.yaml": Ingress in version "v1" cannot be handled as a Ingress: strict decoding error: unknown field "spec.rules[0].http.paths[0].backend.service.namespace"

        backend:
          service:
            name: magellan-nbi

$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/magellan-nbi created

LB creating this time

Frontend IP coming in

warning -1 backend unhealthy

Increase replicas to 2

077d8fa

$ kubectl apply -f deployment.yaml
deployment.apps/magellan-nbi configured

both zones up now

scale replicas to 3

$ kubectl apply -f deployment.yaml
deployment.apps/magellan-nbi configured

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl get pods -n magellan
NAME                            READY   STATUS    RESTARTS   AGE
magellan-nbi-586d658668-5w5zp   1/1     Running   0          8s
magellan-nbi-586d658668-bxnzv   1/1     Running   0          26m
magellan-nbi-586d658668-w95mk   1/1     Running   0          6m29s

deleting and recreating load balancer
$ kubectl delete -f ingress.yaml
ingress.networking.k8s.io "magellan-nbi" deleted from magellan namespace

a couple min

check service is running on one of the pods
michael@cloudshell:~$ kubectl get pods -n magellan
NAME                            READY   STATUS    RESTARTS   AGE
magellan-nbi-586d658668-5w5zp   1/1     Running   0          6m45s
magellan-nbi-586d658668-8675t   1/1     Running   0          54s
magellan-nbi-586d658668-bxnzv   1/1     Running   0          33m
michael@cloudshell:~$ kubectl exec -it magellan-nbi-586d658668-bxnzv -- /bin/bash
Error from server (NotFound): pods "magellan-nbi-586d658668-bxnzv" not found
michael@cloudshell:~$ kubectl exec -it magellan-nbi-586d658668-bxnzv -n magellan -- /bin/bash
root@magellan-nbi-586d658668-bxnzv:/# curl http://localhost:8080/nbi/app/health
OK

above local, below another one of the replicas
michael@cloudshell:~$ kubectl exec -it magellan-nbi-586d658668-bxnzv -n magellan -- /bin/bash
root@magellan-nbi-586d658668-bxnzv:/# curl http://10.1.0.147:8080/nbi/app/health
OK

since the NodePort service is working internally - the external load balancer should pick it up via the Ingress

reapply ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: magellan-nbi
  namespace: magellan
  annotations:
    # If the class annotation is not specified it defaults to "gce".
    #kubernetes.io/ingress.class: "gce"
    spec.ingressClassName: "gce"
spec:
  rules:
  - http:
      paths:
      - path: /*
        pathType: ImplementationSpecific
        backend:
          service:
            name: magellan-nbi
            port:
              number: 8080

micha@p1gen6 MINGW64 /c/wse_github/ObrienlabsDev/gcp-infrastructure-as-code/m1-gke-ingress/kubernetes-gke (main)
$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/magellan-nbi created