[compute-plane] Manage webhook certs with cert-manager

[estroz]

Is this related to a problem? Please describe.

The compute-plane stack deploys three operator releases that each require in-cluster TLS certificates for webhook servers: grove-operator, dynamo-operator, and nvca-operator. Each currently handles cert provisioning differently and inconsistently:

• grove-operator uses certProvisionMode: auto, which runs an embedded OPA cert-rotator that self-signs, writes, and rotates certs inside the Secret grove-webhook-server-cert. The rotator also injects CA bundles directly into Webhook*Configuration resources. This is entirely internal with no external visibility or rotation controls.
• dynamo-operator ships with a first-class webhook.certManager.enabled flag but the stack sets it to false, leaving cert management unspecified and falling back to whatever the chart's default behavior is.
• nvca-operator exposes a webhookConfig block but has no cert-manager integration in the current stack configuration.

This fragmentation means there is no uniform lifecycle for webhook certs across the compute plane: no shared rotation policy, no standard observability, and no consistent way to audit cert expiry or CA bundle injection. The grove auto mode cert-rotator also injects CA bundles into Webhook*Configuration resources at runtime, which conflicts with GitOps reconciliation expectations and makes the injection behavior opaque.

Describe the solution you'd like

Add cert-manager as a dependency in the compute-plane helmfile and wire all three operator releases through it for webhook certificate management.

For grove-operator, switch certProvisionMode from auto to manual. In manual mode the operator skips all cert generation, closes the certs-ready gate immediately, and reads certs directly from disk via the mounted Secret volume.

For dynamo-operator, enable the existing webhook.certManager.enabled: true flag.

For nvca-operator, reading from local files directly with auto-reload must be implemented to support cert-manager.

The helmfile 01-dependencies.yaml.gotmpl should gain a cert-manager release in a dependency stage that runs before all three operator releases. A needs: entry on the cert-manager release should be added to each operator release that consumes it.

Concrete outcomes:

• All webhook certs are issued and rotated by cert-manager Certificate resources.
• Cert expiry is visible via standard cert-manager metrics and Prometheus alerts.
• CA bundle injection into *WebhookConfiguration resources is handled declaratively via cert-manager annotations, not by an in-process rotator mutating cluster state at runtime.
• A single ClusterIssuer (self-signed CA by default, configurable per environment) covers all three operators.

Describe alternatives you've considered

Keep grove in auto mode. The built-in OPA rotator is functional for basic operation but injects CA bundles into Webhook*Configuration resources at runtime, which is invisible to GitOps controllers and interferes with drift detection. It also produces no shared rotation policy or expiry alerting across operators.

Use grove auto mode with webhookServerSecret.enabled: false. This is the current approach and avoids Helm clobbering the auto-generated Secret on upgrade, but it still relies on the internal rotator and does not address the CA injection or observability gaps.

Manually managed secrets via an external secret manager. Pre-populating TLS secrets from Vault or ESO would centralize cert origin but adds rotation complexity and an out-of-band provisioning step that breaks the helmfile's self-contained install story. cert-manager keeps rotation in-cluster.

Per-chart bundled cert-manager subcharts. Some charts bundle cert-manager as an optional dependency. Relying on per-chart bundling risks installing multiple cert-manager instances or requires coordination across releases to avoid CRD conflicts. A single stack-level release is cleaner.

Additional context

Example values for grove in manual mode with cert-manager CA injection:

config:
  server:
    webhooks:
      certProvisionMode: manual
      secretName: grove-webhook-server-cert
webhooks:
  podCliqueSetValidationWebhook:
    annotations:
      cert-manager.io/inject-ca-from: grove-system/grove-webhook-cert
  podCliqueSetDefaultingWebhook:
    annotations:
      cert-manager.io/inject-ca-from: grove-system/grove-webhook-cert
  clusterTopologyValidationWebhook:
    annotations:
      cert-manager.io/inject-ca-from: grove-system/grove-webhook-cert

Example values for dynamo:

webhook:
  failurePolicy: Fail
  certManager:
    enabled: true

A cert-manager Certificate resource targeting secretName: grove-webhook-server-cert in grove-system and the equivalent for dynamo would be applied as part of the cert-manager release or as post-install hooks. A shared ClusterIssuer with a self-signed CA would be the default, overridable per environment.