Optimizing CR processing after Kubernetes CRD controller restart

[lingdie]

I'm working with a Kubernetes Custom Resource Definition (CRD) controller. I've noticed that when the controller restarts, it attempts to reconcile all existing Custom Resources (CRs). This process causes a delay in processing newly created CRs.

My questions are:

1. What are the best practices to mitigate this delay in processing new CRs after a controller restart?

2. Does the controller-runtime library provide any built-in mechanisms to implement a priority queue or similar functionality to prioritize the processing of new CRs?

3. If controller-runtime doesn't offer such functionality, what are some effective ways to implement a priority system within the reconciliation loop?

4. Are there any performance considerations or potential drawbacks to implementing a priority system in a CRD controller?

I'm particularly interested in solutions that can be implemented using Go and the controller-runtime library. Any insights, code examples, or best practices would be greatly appreciated.

[yyewolf]

Hi there, I think the feature you're describing corresponds pretty well to this: kubernetes-sigs/controller-runtime#2374

It's already usable even though it's currently an opt-in feature and we use operators in production with this feature enabled. It allows for "0-downtime" rollout of the operator (in a way that new CR will be reconciled before the existing CRs sync finished) and in our case worked without any other considerations.

Also, in an event handler, where you have the queue, you can add items to the queue with a certain priority as an option.

PS: The feature should become default in the next release according to: kubernetes-sigs/controller-runtime#3332