Introduce compatibility version for Kubernetes control-plane upgrades

[logicalhan]

Enhancement Description

• One-line enhancement description (can be used as a release note):

Introduce compatibility version in Kubernetes components to enhance Kubernetes control-plane upgrades. See safer Kubernetes upgrades for more details.

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/4330-compatibility-versions

• Primary contact (assignee): @logicalhan, @jpbetz, @liggitt

• Responsible SIGs: SIG api-machinery

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.31
  • Beta release target (x.y):
  • Stable release target (x.y):

• Alpha

  • KEP (k/enhancements) update PR(s): KEP-4330: Compatibility versions #4395
  • Code (k/k) update PR(s):
    • apimachinery: API Emulation Versioning kubernetes#122891
    • [WIP] Storage version compatibilty mode kubernetes#122301
    • add emulated-version flag to kube-scheduler to control the feature gate. kubernetes#125769
    • add emulated-version flag to kube-controller-manager to control the feature gate. kubernetes#125778
    • [sample-apiserver] Fix: Use Correct Effective Version for kube kubernetes#125941
  • Docs (k/website) update PR(s):
    • Not needed. See Add doc for CompatibilityVersion. website#47007

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[jeremyrickard]

/sig api-machinery

[jpbetz]

/sig architecture

[jpbetz]

/milestone 1.30

[k8s-ci-robot]

@jpbetz: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone 1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jpbetz]

/milestone v1.30

[johnbelamaric]

/lead-opted-in

[jpbetz]

/label lead-opted-in

[mickeyboxell]

Hello @logicalhan 👋, Enhancements team here.

Just checking in as we approach enhancements freeze today ([02:00 UTC Friday 9th February 2024 / 18:00 PDT Thursday 8th February 2024](https://everytimezone.com/s/1ade3dca)):.

This enhancement is targeting for stage alpha for v1.30 (correct me, if otherwise)

Here's where this enhancement currently stands:

• KEP readme using the [latest template](https://github.com/kubernetes/enhancements/tree/master/keps/NNNN-kep-template) has been merged into the k/enhancements repo.
• KEP status is marked as implementable for latest-milestone: 1.30. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• KEP readme has up-to-date graduation criteria
• KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check [here](https://github.com/kubernetes/community/blob/master/sig-architecture/production-readiness.md#submitting-a-kep-for-production-readiness-approval)).

• Make sure to update your status to implementable in the kep.yaml file.
• Please let me know when this enhancement is

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[mickeyboxell]

Hello 👋, v1.30 Enhancements team here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in {current release}, please file an exception request. Thanks!

• It looks like you still need to update the KEP status to implementable for latest-milestone: 1.30.

[salehsedghpour]

/milestone clear

[jpbetz]

This is due to a clerical error. #4395 was intended to be the KEP PR but we neglected to notice is was still marked as provisional. #4502 fixes this.

[salehsedghpour]

@jpbetz Please make sure that #4502 is being merged and also consider filing an exception request.

[jpbetz]

@jpbetz Please make sure that #4502 is being merged and also consider filing an exception request.

#4502 is merged. Do we need a full exception for a clerical error like this?

[sftim]

I have a suggestion for the title of this KEP.

How about "Backwards compatibility for Kubernetes control-plane"? That's the feature we'd be adding.

[siyuanfoundation]

Updating the progress

• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.31
  • Beta release target (x.y): 1.34
  • Stable release target (x.y):

• Alpha
  • KEP (k/enhancements) update PR(s): KEP-4330: Compatibility versions #4395
  • Code (k/k) update PR(s):
    apimachinery: API Emulation Versioning kubernetes#122891
    add emulated-version flag to kube-scheduler to control the feature gate. kubernetes#125769
    add emulated-version flag to kube-controller-manager to control the feature gate. kubernetes#125778
    [sample-apiserver] Fix: Use Correct Effective Version for kube kubernetes#125941
  • Docs (k/website) update PR(s):
    Not needed. See Add doc for CompatibilityVersion. website#47007
• Beta
  • KEP (k/enhancements) update PR(s):
    KEP-4330: Change API availability to forward compatibility #5038
    KEP-4330: update minCompatibilityVersion range. #5006
  • Code (k/k) update PR(s):
    [Umbrella Issue] Compatibility Version Feature Gate Port kubernetes#126926
    chore: port all non deprecated, non apiserver/controller-manager referenced features to versioned kubernetes#126791
    chore: migrate logging featuregates to versioned kubernetes#129411
    test: update e2e compatibility test to have a n minus param and update to have n-1 and n-2 prow periodic tests test-infra#34177
    verification machinery for compatibility version kubernetes#125032
    feat: Refactors featuregate lifecycle management script kubernetes#129414
  • Docs (k/website) update PR(s):

[neolit123]

q: what would be the k8s project stance if the user wishes to emulate a version of k8s that is out of support? with this proposal going to beta (n-3) the scenario becomes possible.

for example, 1.35 is supported for 3 releases, but 1.36 can emulate 1.35 for two more releases before 1.36 going out of support, and finally 1.37 can emulate 1.35 for one additional release before 1.37 going out of support.

this results in the possibility of a user staying on 1.35 for 2 years / 6 releases, thinking they might be on a supported release.
the KEP does not outline as goals and non-goals any similar LTS topics, such as should an emulated version receive support.
at the same time i do see some mentions of this KEP in the wg-lts agenda document.

[aojea]

I don't think emulate means fully support that version or fully emulate that version, it means enable the codepaths that were running on that version in current version, but there is a lot of other code that changed completely, so support as a "fully supported version" is unrealistic

[pohly]

Should the user be made aware of being out of support when they ask to emulate a release that is too old?

[sftim]

Should the user be made aware of being out of support when they ask to emulate a release that is too old?

Kubernetes code doesn't (currently) know support EOL dates.

[liggitt]

Should the user be made aware of being out of support when they ask to emulate a release that is too old?

There are already internal mechanical bounds on the versions that can be requested to be emulated. We would not change the behavior of 1.32 emulating 1.31 when 1.31 reaches EOL... the 1.32 binaries would continue to function exactly the same way when asked to emulate 1.31

[sdodson]

Seems like support status should follow the binary version rather than the compatibility version, if not then then the flexibility of this feature will be pretty limited.

[liggitt]

Seems like support status should follow the binary version rather than the compatibility version, if not then then the flexibility of this feature will be pretty limited.

Yes, that is how this works

[maxin93]

Hello, I found a problem k3s-io/k3s#11853 related to this feature. In K3S, feature-gate of control components cannot be applied normally.

The possible cause is that the DefaultComponentGlobalsRegistry is a global object, the command of each control component are initialized in parallel, each component will invoke AddFlags(fs *pflag.FlagSet), but the command of apiserver is executed before kcm&scheduler. As a result, componentGlobalsRegistry.featureGatesConfig is overwritten. The reference of this object is used in the Set interface of ColonSeparatedMultimapStringString. Therefore, after the value of featuregate flag of apiserver is set, the other control components invoke func AddFlags(fs *pflag.FlagSet). As a result, the value of this object may be set to empty. As a result, the featuregate printed in the flag is empty.

Emm, can someone help?

[neolit123]

these global vars do seem problematic for the 'binary smashing' that k3s is doing. if you have a pr in mind you could send it. i think there is also potential for other concurrency issues.

[maxin93]

these global vars do seem problematic for the 'binary smashing' that k3s is doing. if you have a pr in mind you could send it. i think there is also potential for other concurrency issues.

I just have an immature idea, can we separate this global var by Component?

[neolit123]

these global vars do seem problematic for the 'binary smashing' that k3s is doing. if you have a pr in mind you could send it. i think there is also potential for other concurrency issues.

I just have an immature idea, can we separate this global var by Component?

deferring to @siyuanfoundation

[siyuanfoundation]

Yes, it is possible to separate this global var by Component. But that would require touching a lot of code because all places uses the globalDefaultFeatureGate when querying feature gate.

We are working on a fix of not resetting the feature values if AddFlags() is called multiple times. kubernetes/kubernetes#130079 (comment). That should fix the bug if there is not conflicting flags between different components. Is that sufficient? @maxin93

[maxin93]

Sorry, let me describe my understanding and let me know if it is correct. The premise of this fix is that different components do not have the same flag. But different components will actually have the same flags (v, feature-gates, metric...), which still conflicts, right?

[siyuanfoundation]

Sorry, let me describe my understanding and let me know if it is correct. The premise of this fix is that different components do not have the same flag. But different components will actually have the same flags (v, feature-gates, metric...), which still conflicts, right?

With the fix in kubernetes/kubernetes#130079, different components can have the same flag, as long they do not conflict with each other, for example setting featureA=true in one component and featureA=false in another.

[maxin93]

I found a new issue that were not addressed in that fix: In the current K3S, the controller and scheduler are launched with goroutine almost at the same time. They will all parse their flag feature-gate into GlobalsRegistry.featureGatesConfig, the value of this global variable ComponentGlobalsRegistry.featureGatesConfig is overwritten by the value of the component that is executed later, and then the controller and scheduler start calling ComponentGlobalsRegistry.Set() and use the same feature gates. As a result, the featuregate of the component that is started first does not take effect.

[liggitt]

In the current K3S, the controller and scheduler are launched with goroutine almost at the same time

The code inside kubernetes/kubernetes structures these commands as separate binaries, as we test and release. If k3s is reworking them into a single binary, working through the assumptions that breaks and keeping up with new assumptions that breaks is the responsibility of the k3s implementation (in this case, something simple like a synchronization point between those goroutine commands after parsing flags and after setting global feature gates might be sufficient for k3s)

We won't try to do things that cause issues in a rework like k3s has done, but we can't reasonably support or block on issues caused by out-of-tree significant restructurings of the implementation either.

[dims]

@maxin93 in addition to what @liggitt said, do you mind working with the k3s folks directly. thanks!

[maxin93]

Yes, I'm trying to work with k3s. Thanks for your suggestion.