Batch tagging controller create/delete tags calls

[shiv-amz]

/kind feature

What this PR does / why we need it:

• Tagging controller runs in single threaded loop which can create backlog in clusters with large number of nodes. Added worker count of 10, which can make progress in parallel.
• Added createTag/deleteTag batcher which batches multiple api calls into a single call with multiple instances.
• Batching is behind option tagging-controller-enable-batching flag.

Special notes for your reviewer:
We have a similar batching for other EC2 calls in Karpenter

Sample logs, tagging multiple instances. 
createTags.go:61] Batched create tags {
  Resources: [
    "i-09c32e7b40efd2a37",
    "i-0b8387fdf8ffb4e21",
    "i-035ccfa88750fc37e",
    "i-0f1f9fd038b4a8953",
    "i-0c97fbe1fb9ad5823"
  ],
  Tags: [{
      Key: "k8s.io/cloud-provider-aws",
      Value: "test-cluster"
    }]
}

deleteTags.go:76] Batched delete tags {
  Resources: [
    "i-01288200c4829f248",
    "i-0dfc2e29029a8d5ce",
    "i-026eb630a8ad49f7b",
    "i-0b4e6e3ef3612c4f5"
  ],
  Tags: [{
      Key: "k8s.io/cloud-provider-aws"
      Value: "test-cluster"
    }]
}

Does this PR introduce a user-facing change?:
NONE

[k8s-ci-robot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign mmerkes for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

This issue is currently awaiting triage.

If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

Hi @shiv-amz. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[cartermckinnon]

What's the rationale for making this configurable? If it's useful, a BatchSize config option seems more flexible for tuning.

[shiv-amz]

Yes, we don't need the enable batching flag. I added this for my load testing, to easily compare results between batching versus single calls. I'll remove this in next rev.

[cartermckinnon]

This seems like a useful config option, similar to --concurrent-node-syncs, --concurrent-service-syncs, etc.

[cartermckinnon]

Why is parallelism of API calls gated by the batching-per-call option? these seem like 2 different concerns

[shiv-amz]

They are related since, with single concurrent thread, batching api calls does not work. If all calls are made by single thread, they won't ever get batched. So to enable batching, parallel threads consuming from workqueue is required.

As pointed above, will make workerQueueCount as configurable, and remove the batching flag.

[cartermckinnon]

There's a lot going on here :)

Is there a simpler way we could implement this? Could a goroutine in the tagging controller poll multiple items off the queue (if they're available) and fire off the request? In Karpenter, it makes sense why we'd need a decoupled "batcher" because controller-runtime doesn't expose the underlying work queue to Reconcile, but we have some more flexibility in this controller

[cartermckinnon]

Chatted offline -- we'll split this up into 2 PR's, one that adds a simple parallelism mechanism (with a worker pool), and we'll keep iterating on the batching approach.