feat: Allow tagging on per-subnet basis

[ghost]

Description

New variables public_subnet_tags_per_subnet, private_subnet_tags_per_subnet, etc. Optional, no effect if not specified. If specified, they are additional tags to apply to each respective public and private subnet.

Extend the ability to allow naming and tagging route tables, EIPs, and NAT Gateways on per-subnet basis as well, add variables for that.

Motivation and Context

Previously, it was only possible to customize arbitrary tags on a per-AZ basis, not a per-subnet basis. You could customize the Name tag on a per-subnet basis but not any other tag.

The VPCs in our environment have several different types of subnets. We would like to tag them differently, e.g. for cost attribution and reference in external Terraform code. Currently this is not possible: subnets can only be disambiguated by VPC, AZ, and visibility.

Breaking Changes

No breaking changes

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects
• I have executed pre-commit run -a on my pull request

This is currently running in production

[ghost]

Testing in our infrastructure with:

Generating expected Terraform plan:

[ghost]

This is now live in our infrastructure. I also added the ability to customize route table tagging.

[ghost]

I've completed the rollout of the new tagging system throughout our infrastructure, using the code from this pull request in production. No problems uncovered.

[kahirokunn]

Awesome!
LGTM!

[jseiser]

I think this behavior is what we are looking for.

We wanted to be able to have 2 sets of private subnets.

1 for our EKS cluster's worker nodes, and 1 for some other random servers that still survive post EKS migration. So we could just target the server private subnets when deploying those specific EC2 Instances

[davepattie]

Ahh, I see now that tags per az is no use when you have more than one private subnet per az

[hpschry]

I'd love to see this merged, this would address our use case as well: We'll need individual tags per sub-net, e.g. to control the assignment of load balancers to specific sub-nets in an EKS scenario, which is controlled by sub-net tags.

[reoring]

Once this Pull Request is merged, I will use it right away

[KevinBrooke]

We have been using this branch in production for 3 weeks without any issues.

[davepattie]

@antonbabenko would you mind taking a look at this PR please, it's a really helpful and thorough addition, thanks

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

keep

[jseiser]

@raxod502-plaid

Any chance you can fix the conflicts, id really love to get this one merged in.

[ghost]

There's not much point in fixing merge conflicts until we get confirmation from a maintainer that they are interested in merging the PR.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[kahirokunn]

i need this PR.

[rarescosma]

We're hitting the same needs for our EKS setup at the moment: need to tag secondary subnets differently to tell karpenter to slowly start moving the nodes towards these subnets.

What are the plans for moving this PR forward?

[andrewmiskell]

I'm running into some of the same issues in our environment as well, needing to tag specific subnets differently due to different uses.

[netomin]

Hi, hitting the same needs here, it would be great to have this PR to move forward ;)

[kahirokunn]

I think so!

[bryantbiggs]

But here, it sounds like you are wanting to name each individual subnet differently

*not just name, but more generically - treat them differently via unique tags

[ghost]

this pattern is due to the fact that we currently do not support a generic - "create as many different subnet groups of your choosing" - its only those pre-defined subnet group types that we support in this module today

Yes, that is the requirement. If you provided a way to create an arbitrary set of subnet groups then it would not be necessary to parameterize tags within a subnet group.

We aren't treating subnets as pets, we simply have more types of subnets than just "private" and "public" (plus the few other special cases in this module).

So that said - if there were a generic sub-module where users could create various different groups of subnets, does this issue become a moot point?

Yes, exactly.

[andrewmiskell]

I think a generic submodule to stamp out as many private/public sets of subnets I want each with it's own set of tags would make this whole thing moot, at least in my use-case.

In my case, I need to different sets of public subnets that have different tags. One tagged with Subnet_Type "public" and another tagged with Subnet_Type "sending" (our use case is very particular and somewhat edge case due to the custom software we use). Currently I'm using the redshift subnets feature with enable_public_redshift enabled in order to do this. Having a more generic way to create subnets without have to "repurpose" the pre-defined subnet types would be very helpful.

[erpel]

So this "what you're trying to do is covered by groups" argument is the first time I understood the resistance. To me the groups always seemed really type based rather than the main differentiation. This has me rethinking how we use the module in some way.

My use case mainly comes from a situation like this:
We have for example 5 subnets that are "private" in type. That's more than one per AZ - which is problematic in some services where only one subnet per AZ is accepted.

Using tags would be how I'd like to set which subnets of the type "private" should be used for EKS control plane (don't hold me to the exact example, I'm going off memory here).
The subnets are otherwise quite the same, the fact that they are separate subnets comes from some organizational history...

[bryantbiggs]

If I look at the recent issues/PRs, I think most of them can be traced back to - more subnet flexibility. Such as:

• Custom amount of NAT gateways #1065
• Using create_multiple_public_route_tables = true only creates an internet gateway route on one route table. #1087
• "Error in function call" when using neither private subnets nor NAT #944
• Pass already created VPC id  #1082
• Error when creating VPC without any private subnets and has NAT gateways #1068
• This module, doesnt integrate with the Network Firewall Module #978
• Enable all Az specific resources to have a {resource_type}_tags_per_az map #984
• Confusing DNS64 behaviour with public subnets #972
• fix: NAT gw route table associations shouldn't be created when enable_nat_gateway=false #1090
• feat: Custom NAT gateways #1066
• feat: Enable the creation of private NAT gateways #1072

(not including those that have since been closed - this should suffice to show the intent)

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[PLeS207]

up

[pseudomorph]

What's the state of this? Is there a path forward here?

[bryantbiggs]

it requires time and resources - nothing planned at the moment

[Chili-Man]

@bryantbiggs @antonbabenko We have a use case for this particular feature at work as well. As other folks have pointed out here in the comments already, we would like to have several private subnets, with different roles. In particular, we want to have our VPC look like this:

Public Subnets
3 public subnets for AWS load balancers and nat gateways.
The AWS EKS load balancer controller will manage the LBs, so it needs to be tagged with kubernetes.io/role/elb

Private Subnets
3 private subnets specifically for EKS. These need to be tagged with kubernetes.io/role/internal-elb for the LBC.
3 additional general purpose private subnets for workloads running outside of EKS. These must not be tagged with kubernetes.io/role/internal-elb , since we don't want the LBC to deploy in those subnets.

Although we could technically tell LBC which subnets to specifically use at via annotations, coordination and passing of this information from Terraform to the applications we deploy on the EKS cluster is burdensome and additional work, especially when that particular configuration information can be abstracted away from the applications via the subnet discovery tags. If we could have passed the specific default subnets we want the LBC to deploy in via the controller command line flags, then I would have less qualms about this, but unfortunately that is not the case.

From reading through the issue here, I understand that there are potential plans to have a generic subnet submodule that would address all of the concerns here, but since there are no concrete plans for that, I'd like to suggest that during this interim, we reconsider implementing this feature. Specifically, I'd like to propose the following:

Scope the change of adding per subnet tags to only the private and public subnet groups that are created from this module. Looking through all of the comments here, the primary concern is with these two types of subnets, since the other subnet types (e.g. redshift, elasticache, etc) are already specific with a use case in mind. This minimizes the surface area of change, so its less likely to cause issues down the road and easier to maintain. We already have the "private_subnet_names" (and similar) variable that allows to customize each subnet name , which is in a similar vein as this feature request, so I'm having trouble understanding why this change would be rejected. EDIT: Looks like someone already implemented it here as described in the following PR: #1139

I'm happy to open up a new PR for this, but I'd like to gauge appetite on this from the maintainers. Looking at the this repos issues and pr history, its been a feature request for years now, and I don't see why we need to continue waiting for the perfect solution for it to happen.

[ghost]

PR has not seen movement for 2 years. Closing as unlikely to be merged.

[ghost]

For what it's worth, this is still running successfully as a load-bearing component of our production infrastructure.

[Chili-Man]

@raxod502-plaid can you please keep it open? Even though there hasn't been much movement on it, doesn't necessarily mean the community doesn't want this feature. If you close the issue, the maintainers are more likely to forget about this

[bogatuadrian]

@bryantbiggs @antonbabenko This functionality would be very useful to us as well.

Our use case is that we require different types of private subnets for different types of EKS nodes (due to security considerations), and we need to tag those subnets accordingly, so we can configure Karpenter to scale certain nodes that use their respective subnets.

I also agree with what @Chili-Man said in a previous post. You could consider this change only for the public/private subnets. Some work in that direction has been done here #1139.

Also to address some previous points in this thread:

1. There are definitely reasons to have different types of private subnets, subnets that wouldn't fall in any of the other categories (intra, database etc.), our use-case, separate security constrains for each subnet, being one (of probably many)
2. Tags are essential for Karpenter discoverability, but also incredibly useful in a IaC & GitOps workflow where it's easier to reference a resource by tags, rather than by (a generated) ID.

EDIT to add: having a way to add custom subnets to the module could be an improvement, but the main appeal of this module is the out-of-the-box configuration for how a "private" subnet is configured. I just want that "private" configuration, but for 2 groups of subnets that will end up being used in different ways down the line, hence needing custom tagging.

[github-actions]

This PR has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this PR will be closed in 10 days

[github-actions]

This PR was automatically closed because of stale in 10 days

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.