Non-disruptive certificate rotation

[guydc]

Description:
Currently, we have MTLS connections for:

• Envoy Gateway <> Envoy
• Envoy <> Rate Limit server

Envoy Gateway generates client and server certificates for all of the above components and typically provides them as mounted secrets to relevant pods. A job that runs in the helm pre-install and pre-upgrade hooks is responsible for rotation.

It is a common security practice to use short-lived certificates that are rotated frequently. In Envoy Gateway, CA certificates and leaf certificates are handled with the same level of security (storage, access, ... ), and should both be rotated.

To support frequent and non-disruptive rotation, the following is required:

• All components are capable of dynamically reloading certificates when they change.
• Certificates are rotated in a backwards-compatible manner. For example, a new trusted CA is added and the old CA is removed only after all components were able to load the latest CA/leaf certificate. This is especially relevant for data-plane components like Rate Limit where client-facing failures may occur if Envoy and Rate Limit are not both synchronized on the latests CA/leaf certificates.

Currently, these requirements are not met:

• Dynamic reloading:
  • Envoy Gateway supports re-reading the TLS Key, Cert and CA Cert for each connection:

    gateway/internal/crypto/cert_load.go

    Line 54 in 2385672

    return loadConfig()

  • Envoy supports dynamic reload of TLS Key, Cert and CA Cert through the SDS path-based reference to mounted secrets:

    gateway/internal/infrastructure/common/proxy_sds.go

    Line 10 in 2385672

    // xDS certificate rotation is supported by using SDS path-based resource files.

  • Rate Limit supports dynamic reload of TLS Key/Cert but not CA: https://github.com/envoyproxy/ratelimit/blob/6a2e8262874f012d08830cc34ba8058e66a33819/src/provider/cert_provider.go#L15
• Backwards compatible changes: Envoy Gateway currently overwrites certificates in-place, meaning the change is not backwards-compatible.

  gateway/internal/cmd/certgen.go

  Line 81 in 2385672

  func outputCertsForKubernetes(ctx context.Context, cli client.Client, cfg *config.Server, certs *crypto.Certificates) error {

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[zhaohuabing]

Hi @guydc thanks for explaining the issue-it's very clear!

From my understanding, here's what we need to do:

• Keep the old CA when rotating certs
• The EG, Envoy, and ratelimit server should load both the old and new CA to ensure the verification won't break in the intermediate transition period.

[kfox1111]

Maybe directly, or indirectly related to this, would be spire support (https://github.com/spiffe/spire). Should this get its own ticket?

spire can push updated certificates to the workload as it rotates things. Typically the certs may be replaced each hour, and the ca's updated daily.

Would love to be able to use spire with the envoy gateway.

[guydc]

Hi @kfox1111, I think this would deserve its own issue. Are you interested in using SPIRE only for the Gateway itself (Envoy, Envoy Gateway, Rate Limit server, ... ), or also for communication with the backends?