✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

mzazrivec · 2025-04-09T19:27:34Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

This pull request implements CRD and a controller for provisioning complete networking infrastructure required to install a ROSA-HCP cluster in AWS. The proposal for this implementation has been described in #5381.

Under the hood, the implementation uses cloudformation stack and a static (i.e. no possibility of customization) cloudformation template from rosa-cli

This pull request depends on openshift/rosa#2904 (now merged).

Quick howto:

$ export ROSA_NETWORK_NAME=rosa-net-01
$ export AWS_REGION=us-west-2
$ export AVAILABILITY_ZONE_COUNT=2
$ export CIDR_BLOCK=10.0.0.0/16
$ clusterctl generate yaml --from templates/rosa-network.yaml > rosa-net-01.yaml
$ kubectl apply -f rosa-net-01.yaml

To use the ROSANetwork from ROSA control plane:

apiVersion: controlplane.cluster.x-k8s.io/v1beta2
kind: ROSAControlPlane
metadata:
  name: rosa-hcp01-control-plane
  namespace: default
spec:
  rosaNetworkRef:
    name: rosa-net01

and skip / remove subnets and availability zones from the CP spec.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

Release note:

New API for provisioning network infrastructure for ROSA clusters

k8s-ci-robot · 2025-04-09T19:27:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andidog for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-04-09T19:27:44Z

Hi @mzazrivec. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

serngawy · 2025-04-10T14:39:00Z

config/crd/patches/webhook_in_rosanetworks.yaml

+    webhookClientConfig:
+      # this is "\n" used as a placeholder, otherwise it will be rejected by the apiserver for being blank,
+      # but we're going to set it later using the cert-manager (or potentially a patch if not using cert-manager)
+      caBundle: Cg==


No need to add the caBundle.

serngawy · 2025-04-10T14:45:18Z

exp/api/v1beta2/rosanetwork_types.go

+	Resource string `json:"resource"`
+
+	// Identified of the created resource. Will be filled in once the resource is created & ready
+	ID string `json:"ID"`


Suggested change

ID string `json:"ID"`

Id string `json:"id"`

Or resourceId

serngawy · 2025-04-10T14:46:01Z

exp/api/v1beta2/rosanetwork_types.go

+// CFResource groups information pertaining to a resource created as a part of a cloudformation stack
+type CFResource struct {
+	// Name of the created resource: NATGateway1, VPC, SecurityGroup, ...
+	Resource string `json:"resource"`


Suggested change

Resource string `json:"resource"`

Name string `json:"name"`

OR resourceName

serngawy · 2025-04-10T14:49:19Z

exp/api/v1beta2/rosanetwork_types.go

+	Status string `json:"status"`
+
+	// Message pertaining to the status of the resource
+	Reason string `json:"reason"`


message is better I guess ?

Suggested change

Reason string `json:"reason"`

Message string `json:"message"`

serngawy · 2025-04-10T14:51:37Z

exp/api/v1beta2/rosanetwork_types.go

+	// Availability zone of the subnet pair
+	AvailabilityZone string `json:"availabilityZone"`
+
+	// ID of the public subnet


Suggested change

// ID of the public subnet

// Public subnet Id ex; subnet-xxxxxxxxxx

serngawy · 2025-04-10T14:53:59Z

main.go

@@ -284,6 +284,15 @@ func main() {
 		}
 	}

+	// TODO: feature gates?


I don't think we need a new feature gate, we can have it under ROSA feature gate

Yes. I did not mean a new feature gate here, just the existing rosa FG.

serngawy · 2025-04-23T11:46:57Z

you also need to update the ValidatingWebhookConfiguration and MutatingWebhookConfiguration here

serngawy

/ok-to-test

serngawy · 2025-04-24T13:43:59Z

exp/api/v1beta2/rosanetwork_types.go

+	// If no identity is specified, the default identity for this controller will be used.
+	//
+	// +optional
+	IdentityRef *infrav1.AWSIdentityReference `json:"identityRef,omitempty"`


Not sure, if we want to provide this option to end user. We don't do that with RosaControlPlane only default aws identity. However, we should provide OCM identityRef

Why shouldn't we provide this option to the end user? We need to specify the ref to the aws secret somehow. Here I'm just reusing existing structures & code.

What do you mean by OCM identity ref? OCM will not be involved here in any way.

well, to use openshift/rosa and establish ocm client you need to have ocm authentication. Is this not the case with the RosaNetwork CF stack creation?

No. No OCM credentials are needed for rosanet, just AWS credentials.

@serngawy Are you satisfied with the answers here?

@mzazrivec I do remember we discuss that, but after checking the ROSANetwork cloud formation stack template , there are tags added as rosa_hcp_policy and roas service here.
Those tags I think is used to check for privileges ?
I think we have to authenticate the ocm credential. Even if we don't need to create the CF stack but enduser must be a valid OCM user.

@serngawy What does creating VPC with certain tags and checking OCM credentials have to do with each other?

okay, as we discussed no need to have ocm authentication.

serngawy · 2025-08-14T17:41:12Z

please add release note, stating add new ROSA network API

mzazrivec · 2025-08-14T18:36:24Z

please add release note, stating add new ROSA network API

Added.

nrb · 2025-08-15T20:53:25Z

exp/api/v1beta2/conditions_consts.go

+
+const (
+	// ROSANetworkReadyCondition condition reports on the successful reconciliation of ROSANetwork.
+	ROSANetworkReadyCondition clusterv1.ConditionType = "ROSANetworkReady"


This isn't a blocker because we'll need to revisit this across the entire project, but conditions will be changing upstream and we'll need to revisit these to make sure they're conforming to the accepted proposal.

Yep, you mentioned in my proposal PR that the Reason field needs to be set also in the case of success, which is what this PR does.

nrb

I have some suggestions for idiomatic Go style that shouldn't affect the functionality of the PR.

Also have some questions about a few of the interface design decisions.

Once those are answered and we have some baseline testing for the ROSANetwork controller, I think we'll be in good shape.

nrb · 2025-08-15T21:01:17Z

exp/controllers/rosanetwork_controller.go

+}
+
+// ROSANetworkFinalizer allows the controller to clean up resources on delete.
+const ROSANetworkFinalizer = "rosanetwork.infrastructure.cluster.x-k8s.io"


Most other packages in the repo put these finalizer definitions in the corresponding api/<version>/<name>_types.go files, but I see there's other ROSA files that put their finalizers in the controller package.

Ideally we'd clean this up, but it's not a blocker.

agree, moving this to the API is better

@nrb yep, moved.

nrb · 2025-08-15T21:02:50Z

exp/controllers/rosanetwork_controller.go

+		if apierrors.IsNotFound(err) {
+			return ctrl.Result{}, nil
+		}
+		return ctrl.Result{Requeue: true}, nil


We do nothing with the error at all? Should we perhaps log it, so the user can at least see that errors are causing reconcile loops?

I'm glad you asked, because I asked this very question within my team last week.

Short answer to your original question would be: this same pattern is in several places in CAPA.

Of course, I can add the error logging there. But probably same thing should have to be done in

controlplane/rosa/controllers/rosacontrolplane_controller.go exp/controllers/awsfargatepool_controller.go exp/controllers/awsmanagedmachinepool_controller.go exp/controllers/rosamachinepool_controller.go

@mzazrivec just return the error here , other controllers can be done in another PR.

nrb · 2025-08-15T21:07:23Z

exp/controllers/rosanetwork_controller.go

+		for i := 1; i <= len(rosaNetScope.ROSANetwork.Spec.AvailabilityZones); i++ {
+			cfParams[fmt.Sprintf("AZ%d", i)] = rosaNetScope.ROSANetwork.Spec.AvailabilityZones[i-1]
+		}


Suggested change

for i := 1; i <= len(rosaNetScope.ROSANetwork.Spec.AvailabilityZones); i++ {

cfParams[fmt.Sprintf("AZ%d", i)] = rosaNetScope.ROSANetwork.Spec.AvailabilityZones[i-1]

}

for i, zone := range rosaNetScope.ROSANetwork.Spec.AvailabilityZones) {

cfParams[fmt.Sprintf("AZ%d", i)] = zone

}

is more Go-native.

@nrb yep, changed

nrb · 2025-08-15T21:12:01Z

exp/controllers/rosanetwork_controller.go

+		for i := 1; i <= len(rosaNetScope.ROSANetwork.Spec.AvailabilityZones); i++ {
+			cfParams[fmt.Sprintf("AZ%d", i)] = rosaNetScope.ROSANetwork.Spec.AvailabilityZones[i-1]
+		}
+		cfTags := map[string]string{}


Should probably have some tags identifying the resources in AWS, in case an administrator is looking at the AWS console and wondering where the CF resources came from.

You can look into how we do it for clusterawsadm, and find a Tag type in the API.

I'm not entirely sure I understand what you mean here.

The particular stack resources (VPC, subnets, ...) are already being tagged as is defined in the CF template from rosa-cli.

If you're talking about tagging the CF stack object itself, then I don't see clusterawsadm bootstrap creating those.

Expose tags field to ROSANetworkSpec so administrator (endUser) can add tags to the created VPC. Its a common use of tags with aws resources.

nrb · 2025-08-15T21:14:15Z

exp/controllers/rosanetwork_controller.go

+			}
+		}
+
+		if resource.LogicalID[0:13] == "SubnetPrivate" {


Let's use https://pkg.go.dev/strings#HasPrefix

Sure, done.

nrb · 2025-08-15T21:18:36Z

exp/controllers/rosanetwork_controller.go

+	i := 0
+	for _, v := range subnets {
+		rosaNet.Status.Subnets[i] = *v
+		i++
+	}


Suggested change

i := 0

for _, v := range subnets {

rosaNet.Status.Subnets[i] = *v

i++

}

rosaNet.Status.Subnets = make([]expinfrav1.ROSANetworkSubnet)

rosaNet.Status.Subnets = append(rosaNet.Status.Subnets, subnets...)

will copy the entries into a new slice. Note that if you specify a length with the make call when doing it this way, you'll see double the number of entries that you'd expect.

It also doesn't look like we're doing anything with i here. If you want the index produced by range, you can do for i, v := range subnets.

Not sure I understand. rosaNet.Status.Subnets should in the end be an array of objects, looking like this:

kind: ROSANetwork metadata: name: rosa-network-01 namespace: default status: subnets: - availabilityZone: us-west-2a privateSubnet: subnet-1d9f28ba992a83514 publicSubnet: subnet-0d9f28ba991b93514 - availabilityZone: us-west-2b privateSubnet: subnet-2d7f58c09f1b43512 publicSubnet: subnet-2d7f18c09f1b43512 - availabilityZone: us-west-2c privateSubnet: subnet-7d7e19c0af1f4d57f publicSubnet: subnet-1d7e19c0af1c4c57f

In my code subnets is a map (subnets := make(map[string]*expinfrav1.ROSANetworkSubnet)), so len(subnets) should return the number of keys, which is what I want. I certainly don't want to append subnets to rosaNet.Status.Subnets, just its values.

@mzazrivec better to use native function to get the map values instead of doing for loop , check here

nrb · 2025-08-15T21:20:20Z

pkg/cloud/scope/rosanetwork.go

+}
+
+// Close closes the current scope persisting the rosanetwork configuration and status.
+func (s *ROSANetworkScope) Close() error {


Why have this extra function if it only calls PatchObject and returns any error it gets?

Yeah. Again, something that all the other scope objects do. I can change it of course, but this is a pattern and if it's not desirable, it should be probably changed everywhere.

nrb · 2025-08-15T21:22:14Z

pkg/cloud/interfaces.go

@@ -104,4 +104,6 @@ type SessionMetadata interface {
 	InfraCluster() ClusterObject
 	// IdentityRef returns the AWS infrastructure cluster identityRef.
 	IdentityRef() *infrav1.AWSIdentityReference
+	// ControllerName returns the controller name
+	ControllerName() string


Why is the controller being exposed here?

Because I need the controller name when constructing key to session cache in pkg/cloud/scope/session.go (see my change in getSessionName() there). Without that change we could hit conflicts in the cache, should for example both the rosanet and rosacp be named the same and be placed in the same namespace.

nrb · 2025-08-15T21:22:53Z

pkg/cloud/scope/rosanetwork_test.go

+	g.Expect(rosaNetScope.Session()).ToNot(BeNil())
+
+	// AWSClusterStaticIdentity
+	rosaNetwork.Spec.IdentityRef.Name = "cluster-static-identity"


Will other identity types be supported in the future?

Yep, all the current identities should work with rosanet.

nrb · 2025-08-15T21:24:46Z

exp/controllers/rosanetwork_controller.go

+limitations under the License.
+*/
+
+package controllers


It would be nice to have some basic tests for the controller.

Yes, I added basic exp/controllers/rosanetwork_controller_test.go, which I want to keep on extending.

k8s-ci-robot · 2025-08-29T13:10:41Z

@mzazrivec: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-aws-apidiff-main	`2c7bf06`	link	false	`/test pull-cluster-api-provider-aws-apidiff-main`
pull-cluster-api-provider-aws-verify	`2c7bf06`	link	true	`/test pull-cluster-api-provider-aws-verify`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot requested review from faiq and serngawy April 9, 2025 19:27

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 9, 2025

mzazrivec force-pushed the rosa_network branch from 3435873 to 22dea0b Compare April 9, 2025 19:32

serngawy reviewed Apr 10, 2025

View reviewed changes

mzazrivec force-pushed the rosa_network branch 4 times, most recently from 5907fb1 to 24a5950 Compare April 24, 2025 13:20

mzazrivec force-pushed the rosa_network branch 3 times, most recently from a947563 to a255790 Compare May 19, 2025 13:43

mzazrivec force-pushed the rosa_network branch from a255790 to d923d6f Compare May 28, 2025 12:17

serngawy reviewed May 28, 2025

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels May 28, 2025

mzazrivec force-pushed the rosa_network branch 4 times, most recently from d2534a7 to dcc599d Compare June 9, 2025 08:19

mzazrivec force-pushed the rosa_network branch from dcc599d to 023f99b Compare June 23, 2025 14:09

mzazrivec force-pushed the rosa_network branch from 5a48f82 to 07513d3 Compare August 14, 2025 16:42

mzazrivec changed the title ~~✨ RosaNetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP~~ ✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP Aug 14, 2025

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Aug 14, 2025

mzazrivec force-pushed the rosa_network branch from 07513d3 to 776f85c Compare August 15, 2025 10:37

nrb reviewed Aug 15, 2025

View reviewed changes

nrb requested changes Aug 15, 2025

View reviewed changes

mzazrivec force-pushed the rosa_network branch from 776f85c to dae6a83 Compare August 18, 2025 12:18

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 19, 2025

RosaNetwork: new CRD & reconciler to provision net infra for ROSA-HCP

56760b6

mzazrivec force-pushed the rosa_network branch from dae6a83 to fe619b4 Compare August 21, 2025 16:36

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 21, 2025

mzazrivec force-pushed the rosa_network branch 12 times, most recently from 6a884a4 to e646926 Compare August 28, 2025 15:18

mzazrivec force-pushed the rosa_network branch from e646926 to fcdd0c0 Compare August 29, 2025 12:54

ROSANetwork: tests

2c7bf06

mzazrivec force-pushed the rosa_network branch from fcdd0c0 to 2c7bf06 Compare August 29, 2025 12:59

	Reason string `json:"reason"`
	Message string `json:"message"`

	// ID of the public subnet
	// Public subnet Id ex; subnet-xxxxxxxxxx

✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

Are you sure you want to change the base?

✨ ROSANetwork: new CRD & reconciler to provision network infrastructure for ROSA-HCP #5464

Conversation

mzazrivec commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 9, 2025

Uh oh!

k8s-ci-robot commented Apr 9, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serngawy commented Apr 23, 2025

Uh oh!

serngawy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serngawy Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

serngawy commented Aug 14, 2025

Uh oh!

mzazrivec commented Aug 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nrb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mzazrivec commented Apr 9, 2025 •

edited

Loading

serngawy Aug 18, 2025 •

edited

Loading