[Bug]: GCP resources not created properly/ cannot be deleted due to conversion webhook TLS not being set correctly

[MarijnMB]

Is there an existing issue for this?

• I have searched the existing issues

It might be related to #667, but I'm not sure if the cause is the same.

Affected Resource(s)

• buckets.storage.gcp.upbound.io
• bucketiammembers.storage.gcp.upbound.io
• projectiammembers.cloudplatform.gcp.upbound.io
• serviceaccounts.cloudplatform.gcp.upbound.io
• serviceaccountkeys.cloudplatform.gcp.upbound.io

And probably more

Resource MRs required to reproduce the bug

Installed crossplane using argocd:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: crossplane
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  sources:
    - chart: crossplane
      repoURL: https://charts.crossplane.io/stable
      targetRevision: 1.19.0
      helm:
        valuesObject:
          provider:
            packages:
              - xpkg.upbound.io/upbound/provider-gcp-storage:v1
              - xpkg.upbound.io/upbound/provider-gcp-cloudplatform:v1

Installed the providerconfig:

apiVersion: gcp.upbound.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  projectID: our-gcp-project-id
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: gcp-secret
      key: creds

Steps to Reproduce

Install crossplane, install gcp provider storage, create a bucket (all according to the 'getting started with GKE' section in the docs). Happens with other resource types too

Bucket resource (also works with a very simple bucket without all the properties):

apiVersion: storage.gcp.upbound.io/v1beta1
kind: Bucket
metadata:
  name: {{ include "backend.bucket.name" . }}
  labels:
    testing.upbound.io/bucket-name: {{ include "common.resourceNamePrefix" . }}bucket
spec:
  forProvider:
    location: EUROPE-WEST4
    autoclass:
      - enabled: true
        terminalStorageClass: ARCHIVE
    cors:
      - maxAgeSeconds: 3600
        method: ["GET", "PUT", "OPTIONS"]
        origin: ["*"]
        responseHeader: ["Accept", "Authorization", "Content-Range", "Content-Type", "Origin", "Range"]
    defaultEventBasedHold: false
    forceDestroy: true
    storageClass: STANDARD
    publicAccessPrevention: inherited
    requesterPays: false
    uniformBucketLevelAccess: true
    versioning:
      - enabled: false
  providerConfigRef:
    name: default

What happened?

Resource (bucket) is created, but creation status is not updated. bucket also cannot be removed by removing the bucket resource.

Relevant Error Output Snippet

$ kubectl get buckets -v=9
I0217 12:51:42.930935   37679 loader.go:395] Config loaded from file:  /home/marijn/.kube/config
I0217 12:51:42.936875   37679 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b" 'https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500'
I0217 12:52:12.957139   37679 round_trippers.go:553] GET https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500 500 Internal Server Error in 30020 milliseconds
I0217 12:52:12.957184   37679 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 30020 ms Duration 30020 ms
I0217 12:52:12.957197   37679 round_trippers.go:577] Response Headers:
I0217 12:52:12.957211   37679 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 373e935e-c497-45f4-8f8e-a754d07837eb
I0217 12:52:12.957221   37679 round_trippers.go:580]     Content-Length: 366
I0217 12:52:12.957230   37679 round_trippers.go:580]     Date: Mon, 17 Feb 2025 11:52:12 GMT
I0217 12:52:12.957238   37679 round_trippers.go:580]     Audit-Id: 1d7f54f7-c89a-45d6-a0f2-218aa8b8a0ac
I0217 12:52:12.957247   37679 round_trippers.go:580]     Cache-Control: no-cache, private
I0217 12:52:12.957255   37679 round_trippers.go:580]     Content-Type: application/json
I0217 12:52:12.957264   37679 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 96c33677-45dd-4b0a-8a86-1e437da71c3a
I0217 12:52:12.957303   37679 request.go:1212] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"conversion webhook for storage.gcp.upbound.io/v1beta1, Kind=Bucket failed: Post \"https://upbound-provider-gcp-storage.crossplane-system.svc:9443/convert?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)","code":500}
I0217 12:52:12.957776   37679 helpers.go:246] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "conversion webhook for storage.gcp.upbound.io/v1beta1, Kind=Bucket failed: Post \"https://upbound-provider-gcp-storage.crossplane-system.svc:9443/convert?timeout=30s\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)",
  "code": 500
}]
Error from server: conversion webhook for storage.gcp.upbound.io/v1beta1, Kind=Bucket failed: Post "https://upbound-provider-gcp-storage.crossplane-system.svc:9443/convert?timeout=30s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Manually trying the curl command listed in the output:

$ curl -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b" 'https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500'
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.0.11.2:443...
* Connected to 10.0.11.2 (10.0.11.2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Same command, but telling curl to ignore insecure tls (-k flag):

$ curl -k -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b" 'https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500'
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.0.11.2:443...
* Connected to 10.0.11.2 (10.0.11.2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=34.91.139.5
*  start date: Apr 19 01:23:46 2024 GMT
*  expire date: Apr 18 01:25:46 2029 GMT
*  issuer: CN=e70a48ea-049f-4334-a2df-3a8196ccb309
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 10.0.11.2]
* [HTTP/2] [1] [:path: /apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500]
* [HTTP/2] [1] [accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json]
* [HTTP/2] [1] [user-agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b]
> GET /apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500 HTTP/2
> Host: 10.0.11.2
> Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
> User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* received GOAWAY, error=0, last_stream=1
< HTTP/2 403
< audit-id: a0e5f131-7d56-4643-86c7-94fbe52a5ba5
< cache-control: no-cache, private
< content-type: application/json
< x-content-type-options: nosniff
< x-kubernetes-pf-flowschema-uid: 96c33677-45dd-4b0a-8a86-1e437da71c3a
< x-kubernetes-pf-prioritylevel-uid: 373e935e-c497-45f4-8f8e-a754d07837eb
< content-length: 340
< date: Mon, 17 Feb 2025 11:54:12 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"buckets.storage.gcp.upbound.io is forbidden: User \"system:anonymous\" cannot list resource \"buckets\" in API group \"storage.gcp.upbound.io\" at the cluster scope","reason":"Forbidden","details":{"group":"storage.gcp.upbound.io","kind":"buckets"},"code":403}
* Closing connection
* TLSv1.3 (OUT), TLS alert, close notify (256):

Forbidden is expected here, as the curl command isn't authed without the kubectl wrapper.

Checking the certs returned by the server/webhook endpoint:

$ openssl s_client -connect 10.0.11.2:443 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = 34.91.139.5
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = 34.91.139.5
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = 34.91.139.5
verify return:1
---
Certificate chain
 0 s:CN = 34.91.139.5
   i:CN = e70a48ea-049f-4334-a2df-3a8196ccb309
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 19 01:23:46 2024 GMT; NotAfter: Apr 18 01:25:46 2029 GMT
-----BEGIN CERTIFICATE-----
MIIEIjCCAoqgAwIBAgIQKdcpM6Qt9kGJK8u3GZTGzTANBgkqhkiG9w0BAQsFADAv
<snipped for security>
8MfsAGmU
-----END CERTIFICATE-----
---
Server certificate
subject=CN = 34.91.139.5
issuer=CN = e70a48ea-049f-4334-a2df-3a8196ccb309
---
Acceptable client certificate CA names
CN = e70a48ea-049f-4334-a2df-3a8196ccb309
CN = 4dc83e93-311f-4bf0-9b03-2f04da2bad02
Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1777 bytes and written 387 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_128_GCM_SHA256
    Session-ID: AA8D5E1E004954E916E6C790F954D0195590F2677F08D03C718743FA4587E60D
    Session-ID-ctx:
    Resumption PSK: 45511D96D4DE08EA990BF2138F82026819CE16AF0B70F8AB03346283FBC65426
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 604800 (seconds)
    TLS session ticket:
    0000 - 0c a5 e9 51 d0 16 fc f2-42 99 c4 b6 24 bd fe 04   ...Q....B...$...
<snip>
    0060 - 63 45 0b d2 df e5 33 19-b4                        cE....3..

    Start Time: 1739793303
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
    Max Early Data: 0

This returned certificate is in NONE of the cert secrets in the crossplane-system namespace. If this cert is saved to file (/tmp/cert2.file), and curl is retried specifying this cert:

$ curl --cacert /tmp/cert2.file -v -XGET  -H "Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json" -H "User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b" 'https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500'
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 10.0.11.2:443...
* Connected to 10.0.11.2 (10.0.11.2) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /tmp/cert2.file
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=34.91.139.5
*  start date: Apr 19 01:23:46 2024 GMT
*  expire date: Apr 18 01:25:46 2029 GMT
*  subjectAltName: host "10.0.11.2" matched cert's IP address!
*  issuer: CN=e70a48ea-049f-4334-a2df-3a8196ccb309
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://10.0.11.2/apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: 10.0.11.2]
* [HTTP/2] [1] [:path: /apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500]
* [HTTP/2] [1] [accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json]
* [HTTP/2] [1] [user-agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b]
> GET /apis/storage.gcp.upbound.io/v1beta2/buckets?limit=500 HTTP/2
> Host: 10.0.11.2
> Accept: application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json
> User-Agent: kubectl/v1.30.9 (linux/amd64) kubernetes/334974b
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* received GOAWAY, error=0, last_stream=1
< HTTP/2 403
< audit-id: 20ab98bf-b829-4021-9cd0-9018fc0dce06
< cache-control: no-cache, private
< content-type: application/json
< x-content-type-options: nosniff
< x-kubernetes-pf-flowschema-uid: 96c33677-45dd-4b0a-8a86-1e437da71c3a
< x-kubernetes-pf-prioritylevel-uid: 373e935e-c497-45f4-8f8e-a754d07837eb
< content-length: 340
< date: Mon, 17 Feb 2025 11:56:56 GMT
<
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"buckets.storage.gcp.upbound.io is forbidden: User \"system:anonymous\" cannot list resource \"buckets\" in API group \"storage.gcp.upbound.io\" at the cluster scope","reason":"Forbidden","details":{"group":"storage.gcp.upbound.io","kind":"buckets"},"code":403}
* Closing connection
* TLSv1.3 (OUT), TLS alert, close notify (256):

Again, forbidden is expected here because the curl isn't wrapped by kubectl.

My current workaround is running this when everything hangs, but that's not really a long-term solution.

kubectl patch crd buckets.storage.gcp.upbound.io -p '{"spec":{"conversion": null}}' --type=merge
kubectl patch crd bucketiammembers.storage.gcp.upbound.io -p '{"spec":{"conversion": null}}' --type=merge
kubectl patch crd projectiammembers.cloudplatform.gcp.upbound.io -p '{"spec":{"conversion": null}}' --type=merge
kubectl patch crd serviceaccounts.cloudplatform.gcp.upbound.io -p '{"spec":{"conversion": null}}' --type=merge
kubectl patch crd serviceaccountkeys.cloudplatform.gcp.upbound.io -p '{"spec":{"conversion": null}}' --type=merge

Crossplane Version

1.19.0

Provider Version

v1 (1.14.4)

Kubernetes Version

v1.30.8-gke.1162001

Kubernetes Distribution

GKE

Additional Info

No response