adding mtls config #1170

laurafitzgerald · 2025-02-20T13:10:54Z

Changes

Verification steps
Run the following on this branch.
make local-setup
kubectl apply -f config/samples/kuadrant_v1beta1_kuadrant.yaml
Follow steps in https://docs.kuadrant.io/dev/kuadrant-operator/doc/user-guides/full-walkthrough/secure-protect-connect/
Update the kuadrant cr to enable mts using
kubectl patch kuadrant kuadrant-sample --type='json' -p='[{"op": "replace", "path": "/spec/mtls/enable", "value": true}]'
Verify that a Peer Authentication exists
kubectl get peerauthentication default -n default -o yaml
which has the following details

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  labels:
    kuadrant.io/managed: "true"
  name: default
  namespace: default
spec:
  mtls:
    mode: STRICT

Verify that the two envoyfilters have a CLUSTER patch with a key "transport_socket" in spec.configPatches applyTo CLUSTER.patch.value
kubectl get envoyfilter -A -o yaml

Verify that the limitador and authorino deployments have the expected labels
kubectl get deployment limitador-limitador -o yaml | yq e '.spec.template.metadata.labels' kubectl get deployment authorino -o yaml | yq e '.spec.template.metadata.labels'
Both should output

authorino-resource: authorino
control-plane: controller-manager
sidecar.istio.io/inject: "true

Boomatang

There is a few comments that I have. I didn't try running this locally.

I would like to point out with the state of the world and the topology any time you find your self request data from the cluster you are doing something wrong

controllers/mtls_reconciler.go

controllers/state_of_the_world.go

pkg/limitador/conditions.go

Boomatang · 2025-02-20T16:21:39Z

pkg/reconcilers/deployment.go

+func MergeMapStringString(modified *bool, existing *map[string]string, desired map[string]string) {
+	if *existing == nil {
+		*existing = map[string]string{}
+	}
+
+	// for each desired key value set, e.g. labels
+	// check if it's present in existing. if not add it to existing.
+	// e.g. preserving existing labels while adding those that are in the desired set.
+	for desiredKey, desiredValue := range desired {
+		if existingValue, exists := (*existing)[desiredKey]; !exists || existingValue != desiredValue {
+			(*existing)[desiredKey] = desiredValue
+			*modified = true
+		}
+	}
+}


Could we use labels.Set.Merge from the apimachinery to do this over defining our own.

it's not a map merge operation. It's doing what the comments say.

Boomatang · 2025-02-20T16:23:22Z

controllers/mtls_reconciler.go

+		err = r.ReconcileResource(eventCtx, &v12.Deployment{}, deployment, reconcilers.DeploymentMutator(deploymentMutators...))
+	}
+
+	valueMap := map[string]interface{}{


Everything here being a map[sting]interface seems like a code smell.

I believe this is the standard way that EnvoyFilter Config Object Patches arehandle. You can see an exmaple of this here https://github.com/Kuadrant/kuadrant-controller/blob/22535aa8e4f142bf07a69dc6391c84a59b232255/pkg/istio/envoy_filters.go#L14-L47

Boomatang · 2025-03-04T12:47:29Z

pkg/limitador/conditions.go

+
+func IsLimitadorOperatorInstalled(restMapper meta.RESTMapper, logger logr.Logger) (bool, error) {
+	if ok, err := utils.IsCRDInstalled(restMapper, kuadrantv1beta1.LimitadorGroupKind.Group, kuadrantv1beta1.LimitadorGroupKind.Kind, limitadorv1alpha1.GroupVersion.Version); !ok || err != nil {
+		logger.V(1).Error(err, "Limitador Operator CRD was not installed", "group", kuadrantv1beta1.AuthorinoGroupKind.Group, "kind", kuadrantv1beta1.AuthorinoGroupKind.Kind, "version", authorinooperatorv1beta1.GroupVersion.Version)


Might want to fix the references to the authorino types here.

Boomatang

I think this is now at the stage where we need to be able to compile and run the operator to do a good review. It is getting close.

Boomatang · 2025-03-07T09:32:50Z

api/v1beta1/kuadrant_types.go

+type MTLS struct {
+	Enable bool `json:"enable,omitempty"`
+}
+


At some point can you add a comment to what the mtls struct is for. The comment gets added to the CRD definition as a description. You will also need to run make manifest , bundle and helm at some point.

controllers/mtls_reconciler.go

Signed-off-by: Laura Fitzgerald <[email protected]>

…ation Signed-off-by: Laura Fitzgerald <[email protected]>

…ts to limitador and authorino crs Signed-off-by: Laura Fitzgerald <[email protected]>

…e other cleanup and it's runnable now Signed-off-by: Laura Fitzgerald <[email protected]>

Signed-off-by: Laura Fitzgerald <[email protected]>

laurafitzgerald · 2025-03-12T15:10:33Z

controllers/mtls_reconciler.go

+	if kObj == nil && kObj.Spec.MTLS == nil && (!kObj.Spec.MTLS.Enable || anyAttachedAuthorRateLimitPolicy == false) {
+		defer logger.V(1).Info("mtls not enabled or applicable", "status", "completed")
+		peerAuthentications := lo.FilterMap(topology.Objects().Items(), func(item machinery.Object, _ int) (machinery.Object, bool) {
+			if peerAuthentication, ok := item.(*PeerAuthentication); ok {


This isn't working yet. It's not picking up the exising PeerAuthentications.

eguzki

Very good progress.

Few questions:

If the gateway provider is not Istio and a user enables mtls, should we report in kuadrant status?
When a user deletes a gateway, the peerAuthentications objects gets deleted? (not very important now and can be tackled in follow up PR's)

eguzki · 2025-03-14T14:58:27Z

api/v1beta1/topology.go

+		To:   DeploymentGroupKind,
+		Func: func(child machinery.Object) []machinery.Object {
+			return lo.Filter(authorinos, func(k machinery.Object, _ int) bool {
+				return k.GetNamespace() == child.GetNamespace() && child.GetName() == "authorino"


I am not very familiar with these linking functions. I still need to educate myself.

Question: child's type is Authorino, isn't it? Then, isn't this linking function linking all deployments in Authorino namespace to Authorino? even those that are not authorino deployment, like limitador deployment?

eguzki · 2025-03-14T15:00:01Z

config/samples/kuadrant_v1beta1_kuadrant.yaml

@@ -3,4 +3,6 @@ apiVersion: kuadrant.io/v1beta1
 kind: Kuadrant
 metadata:
  name: kuadrant-sample
-spec: {}
+spec:


This is the sample being shown in operatorhub web console. Do we want to add (optional) mtls config to it?

eguzki · 2025-03-14T15:05:13Z

controllers/auth_workflow_helpers.go

+					"tls_certificate_sds_secret_configs": []interface{}{
+						map[string]interface{}{
+							"name": "default",
+							"sds_config": map[string]interface{}{


Am I correct assuming that the gateway will fetch the client certificate from XDS interface?

eguzki · 2025-03-14T15:12:33Z

controllers/istio_auth_cluster_reconciler.go

@@ -56,6 +56,10 @@ func (r *IstioAuthClusterReconciler) Reconcile(ctx context.Context, _ []controll
 	if kuadrant == nil {
 		return nil
 	}
+	mtls := false
+	if kuadrant != nil && kuadrant.Spec.MTLS != nil && kuadrant.Spec.MTLS.Enable {


nit: when this kind of checks depend entirely on an object internal state and can be reused, they can be implemented in the object

diff --git a/api/v1beta1/kuadrant_types.go b/api/v1beta1/kuadrant_types.go index 12bb1545..10316535 100644 --- a/api/v1beta1/kuadrant_types.go +++ b/api/v1beta1/kuadrant_types.go @@ -52,6 +52,14 @@ func (p *Kuadrant) GetLocator() string { return machinery.LocatorFromObject(p) } +func (p *Kuadrant) IsMTLSEnabled() bool { + if p == nil { + return false + } + + return p.Spec.MTLS != nil && p.Spec.MTLS.Enable +} + // KuadrantSpec defines the desired state of Kuadrant type KuadrantSpec struct { Observability Observability `json:"observability,omitempty"`

It works even when kuadrant object is nil.

eguzki · 2025-03-14T15:15:52Z

controllers/mtls_reconciler.go

+	targetables := topology.Targetables()
+	gateways := targetables.Items(func(o machinery.Object) bool {
+		gateway, ok := o.(*machinery.Gateway)
+		return ok && gateway.Spec.GatewayClassName == "istio"


It's better to check the GatewayClass instead of that. Check out how the observability reconciler is doing this https://github.com/Kuadrant/kuadrant-operator/blob/main/controllers/observability_reconciler.go#L399-L422

eguzki · 2025-03-14T15:23:14Z

pkg/istio/utils.go

+		}
+	}
+*/
+func LinkDeploymentToAuthorino(objs controller.Store) machinery.LinkFunc {


Isn't this defined in https://github.com/Kuadrant/kuadrant-operator/pull/1170/files#diff-88007d2400b1fe3a9b5d5d7f78bd367e3721cadcddc40033dd9170fd040dba3fR70 ?

eguzki · 2025-03-14T15:24:43Z

pkg/limitador/conditions.go

+	"k8s.io/apimachinery/pkg/api/meta"
+)
+
+func IsLimitadorOperatorInstalled(restMapper meta.RESTMapper, logger logr.Logger) (bool, error) {


Why is required to check the existence of the operator? Shouldn't it be Limitador itself?

eguzki · 2025-03-14T15:27:22Z

pkg/reconcilers/deployment.go

+func MergeMapStringString(modified *bool, existing *map[string]string, desired map[string]string) {
+	if *existing == nil {
+		*existing = map[string]string{}
+	}
+
+	// for each desired key value set, e.g. labels
+	// check if it's present in existing. if not add it to existing.
+	// e.g. preserving existing labels while adding those that are in the desired set.
+	for desiredKey, desiredValue := range desired {
+		if existingValue, exists := (*existing)[desiredKey]; !exists || existingValue != desiredValue {
+			(*existing)[desiredKey] = desiredValue
+			*modified = true
+		}
+	}
+}


it's not a map merge operation. It's doing what the comments say.

laurafitzgerald requested a review from a team as a code owner February 20, 2025 13:10

laurafitzgerald requested a review from Boomatang February 20, 2025 13:11

laurafitzgerald marked this pull request as draft February 20, 2025 13:12

Boomatang reviewed Feb 20, 2025

View reviewed changes

laurafitzgerald force-pushed the mtls branch from 7b55e2a to 83a6d54 Compare February 26, 2025 15:16

Boomatang reviewed Mar 4, 2025

View reviewed changes

Boomatang reviewed Mar 7, 2025

View reviewed changes

KevFan reviewed Mar 10, 2025

View reviewed changes

controllers/mtls_reconciler.go Outdated Show resolved Hide resolved

KevFan reviewed Mar 10, 2025

View reviewed changes

controllers/mtls_reconciler.go Outdated Show resolved Hide resolved

laurafitzgerald force-pushed the mtls branch 3 times, most recently from da6de62 to e64e213 Compare March 12, 2025 12:45

laurafitzgerald marked this pull request as ready for review March 12, 2025 12:45

laurafitzgerald added 8 commits March 12, 2025 12:47

adding mtls config

daa0277

Signed-off-by: Laura Fitzgerald <[email protected]>

move istio check to before the reconciler

1caf65d

Signed-off-by: Laura Fitzgerald <[email protected]>

move limitador and authorino installed checks to reconciler initialis…

ff34a16

…ation Signed-off-by: Laura Fitzgerald <[email protected]>

refactor to add some cleanup logic and adding logic to link deploymen…

61e57f1

…ts to limitador and authorino crs Signed-off-by: Laura Fitzgerald <[email protected]>

moving envoyfilter management to auth and rate limit reconcilers, som…

f6a27ce

…e other cleanup and it's runnable now Signed-off-by: Laura Fitzgerald <[email protected]>

cleanup of restmapper and variables

e68fe2e

Signed-off-by: Laura Fitzgerald <[email protected]>

rebase and fixup and imports

4c6f0f4

Signed-off-by: Laura Fitzgerald <[email protected]>

updating bundle and chart

87516f4

Signed-off-by: Laura Fitzgerald <[email protected]>

laurafitzgerald force-pushed the mtls branch from e64e213 to 87516f4 Compare March 12, 2025 12:48

update sample kuadrant to default to not enable mtls

45de713

Signed-off-by: Laura Fitzgerald <[email protected]>

laurafitzgerald commented Mar 12, 2025

View reviewed changes

eguzki reviewed Mar 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding mtls config #1170

adding mtls config #1170

laurafitzgerald commented Feb 20, 2025 •

edited

Loading

Boomatang left a comment

Boomatang Feb 20, 2025

eguzki Mar 14, 2025

Boomatang Feb 20, 2025

laurafitzgerald Feb 26, 2025

Boomatang Mar 4, 2025

Boomatang left a comment

Boomatang Mar 7, 2025

laurafitzgerald Mar 12, 2025

eguzki left a comment

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

eguzki Mar 14, 2025

adding mtls config #1170

Are you sure you want to change the base?

adding mtls config #1170

Conversation

laurafitzgerald commented Feb 20, 2025 • edited Loading

Boomatang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Boomatang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eguzki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laurafitzgerald commented Feb 20, 2025 •

edited

Loading