KEP-4330: Change API availability to forward compatibility #5038

siyuanfoundation · 2025-01-14T00:57:51Z

One-line PR description: change api availability to forward compatible to make it easier to handle API graduation in compatibility version.

Issue link: Introduce compatibility version for Kubernetes control-plane upgrades #4330

Other comments: more discussions can be found in [Compatibility Version] Guidance on API version promotion kubernetes#127791

k8s-ci-robot · 2025-01-14T00:57:57Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: siyuanfoundation
Once this PR has been reviewed and has the lgtm label, please assign dims for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

keps/sig-architecture/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

siyuanfoundation · 2025-01-14T01:00:37Z

/cc @jpbetz @Jefftree @liggitt @aaron-prindle

keps/sig-architecture/4330-compatibility-versions/README.md

Jefftree · 2025-01-14T15:27:31Z

keps/sig-architecture/4330-compatibility-versions/README.md

+Instead, we are proposing to keep forward compatibility of all AIPs in compatibility version mode: 
+All APIs group-versions declarations would be modified
+to track which Kubernetes version the API group-versions are removed at. APIs introduced between the emulation version and 
+the binary version are still available if the stability level is no less than the APIs introduced before or at the emulation version and enabled at the emulation version.


Most of this is referring to API enablement/disablement on a group-version level. However, API lifecycles are generally individual on the group-version-resource level. I think it would make more sense to discuss the individual beta->GA resources rather than the entire group version?

Jefftree · 2025-01-14T15:42:55Z

keps/sig-architecture/4330-compatibility-versions/README.md


 Alpha APIs may not be enabled in conjunction with emulation version.

+The forward compatibility of APIs should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`.


I think we should add a section on how MinCompatibilityVersion will change storage version since the policy has always been n-1.

Updated in #5006

Jefftree · 2025-01-14T15:49:39Z

keps/sig-architecture/4330-compatibility-versions/README.md

 to track which Kubernetes version the API group-versions are introduced (or
-removed) at.
+removed) at. But in practice, that would make code changes of API graduation intractable. 


intractable only for withholding API introductions. API removals don't need special handling.

Updated the whole section. Please review again see if it is more clear.

Jefftree · 2025-01-14T15:53:50Z

keps/sig-architecture/4330-compatibility-versions/README.md

@@ -610,21 +623,25 @@ The storage version of each group-resource is the newest

 ### API availability

-Similar to feature flags, all APIs group-versions declarations will be modified
+Ideally, similar to feature flags, all APIs group-versions declarations should be modified


Is this referring to the prerelease lifecycle files or a different mechanism? prerelease lifecycle operates on the types instead of the group version.

it is referring to the the prerelease lifecycle.

keps/sig-architecture/4330-compatibility-versions/README.md

jpbetz · 2025-01-14T18:24:38Z

keps/sig-architecture/4330-compatibility-versions/README.md

+removed) at: if an API is introduced after (or removed before) the emulation version, it should not be available at the emulation version. But in practice, that would make code changes of API graduation that wants to use newer APIs intractable. 
+For example, if a controller is switching from Beta to GA API at the binary version, a full emulation would mean the whole controller code needs to be


My understanding is that we're doing this specifically for controllers. Are there other use cases or should we simplify the wording here and just state up front that we're doing this to support controllers specifically?

jpbetz · 2025-01-14T18:25:58Z

keps/sig-architecture/4330-compatibility-versions/README.md

-the emulation version is set to. I.e. `--runtime-config=api/<version>` needs
-to be able to turn on APIs exactly like it did for each Kubernetes version that
-emulation version can be set to.
+If an API has GAed and has not been removed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API would also be enabled at the emulation version along with the old GA API.


A comparison (maybe in a table), or how emulation works for normal APIs vs how it works for un-emulatable (non-emulatable?) features might make this information easier to digest.

jpbetz · 2025-01-14T18:28:10Z

keps/sig-architecture/4330-compatibility-versions/README.md


-GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version
-the emulation version is set to.
+Instead, we are proposing to keep forward compatibility of all APIs in compatibility version mode: 


We're proposing that non-emulatable feature controllers to update to use the newest available API, ignoring emulation API enablement, right? Maybe say this more directly?

Yes, mainly for controllers for now. May expand to kublet if we want to add compat version in node. Updated the section.

keps/sig-architecture/4330-compatibility-versions/README.md

jpbetz · 2025-01-16T17:22:03Z

keps/sig-architecture/4330-compatibility-versions/README.md

+
+With the `NonEmulatableFeatures` list, the server would fail to start if
+1. the `EmulationVersion != BinaryVersion` and
+1. any feature in the `NonEmulatableFeatures` list is disabled by default at the emulation version and explicitly enabled by the `--feature-gates` flag.


What alternatives were considered here? It's not clear to me WHY we're deciding to refuse to start apiservers that have a non-emulatable beta on vs. allowing the beta to be turned on but not guaranteeing emulation for the feature.

Having the apiserver refuse to start is a potential footgun for anyone with automation that uses emulation version, and also offers the capability to turn on beta features/APIs.

Giving more thoughts to it, I think the alternative is the better way to go. Updated the section with pros and cons.

keps/sig-architecture/4330-compatibility-versions/README.md

siyuanfoundation · 2025-01-16T20:59:24Z

/cc @deads2k @johnbelamaric

aojea · 2025-01-20T19:46:44Z

keps/sig-architecture/4330-compatibility-versions/README.md

+```
+To fully emulate the controller for an older version, anywhere v1 api/type is referenced, it would need to switch to the v1beta version if the emulation version is older than the binary version. This would mean a lot of extra work, complicated testing rules, and high maintenance cost even for simple API graduations, while the emulation fidelity is unreliable with the extra complexity.
+
+So instead of truly emulating the feature controllers and API availability at the emulation version, we are proposing to keep forward compatibility of all APIs in compatibility version mode and have the non-emulatable feature controllers to update to use the newest available API. 


Let's say we add or remove field when we graduate from v1beta1 in 1.32 to v1 in 1.33, we may not be able to emulate in the 1.33 binary the behavior of the 1.32 controller ...

If we add a v1 in 1.33, we still need to support clients using v1beta1. So we still should have the code and thus be able to emulate a "v1beta1 only" mode of operation.

One implication of this KEP is probably that we need to keep supporting older versions and behavior for longer than before, but I haven't looked into the details of that yet and could be wrong.

So if a 1.32 controller does thing/A. A 1.33 controller does thing/A and also does wrongThing/B. Does this meant that switching to an emulation version for 1.32 does not stop the 1.33 controller from doing wrongThing/B?

one thing is to support fields and other support the behaviors associated to those fields, what I'm trying to point out is that the controller in 1.33 has a different behavior than the one 1.32, that will not now to use this new field ... however, this is solved with this https://docs.google.com/document/d/1eLulBZ988bWNwrwnz9V70xVbQmHOYjrQpXRwiGmgcFM/edit?disco=AAABYEhCDY0 , if beta to GA does not have change on behavior

there is a difference between supporting fields in APIs and supporting the behaviors. API field changes should always be backward compatible to have data compatibility in storage (https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api_changes.md).

For behavior changes in the code itself, unless we have versioned controllers, it is always best effort: we can either feature gate the new changes, or follow the examples in https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/4330-compatibility-versions#feature-gating-changes. Behavior changes generally are localized, and should be doable with a limited number of if ... else ... switches. If it is big changes that touch a lot of codes, should it really be done for a beta API?

So yes, it is possible if a 1.32 controller does thing/A. A 1.33 controller does thing/A and also does wrongThing/B. switching to an emulation version for 1.32 does not stop the 1.33 controller from doing wrongThing/B if the code to do wrongThing/B is not properly gated based on emulation version.

The proposed API availability changes is meant to only solve the API endpoint problem that is prevalent in API graduations.

keps/sig-architecture/4330-compatibility-versions/README.md

aojea · 2025-01-20T19:52:29Z

keps/sig-architecture/4330-compatibility-versions/README.md

+`v1: introduced=1.28, removed=1.32`<br>`v2beta1: introduced=1.31, removed=1.32`<br>`v2: introduced=1.32` | 1.30 | `api/v1`
+`v1: introduced=1.28, removed=1.32`<br>`v2beta1: introduced=1.31, removed=1.32`<br>`v2: introduced=1.32` | 1.31 | `api/v1` always available, `api/v2beta1` and `api/v2` available only when `--runtime-config=api/v2beta1=true`
+`v1: introduced=1.28, removed=1.32`<br>`v2beta1: introduced=1.31, removed=1.32`<br>`v2: introduced=1.32` | 1.33 | `api/v2`


we don't remove v1 apis, https://kubernetes.io/docs/reference/using-api/deprecation-policy/#deprecating-parts-of-the-api

Rule #4a: API lifetime is determined by the API stability level
GA API versions may be marked as deprecated, but must not be removed within a major version of Kubernetes

updated the table

aojea · 2025-01-20T20:03:07Z

keps/sig-architecture/4330-compatibility-versions/README.md

-the emulation version is set to. I.e. `--runtime-config=api/<version>` needs
-to be able to turn on APIs exactly like it did for each Kubernetes version that
-emulation version can be set to.
+For the use case of upgrading control plane binary version without changing the emulation version, this would mean api servers should be upgraded first before any other components, because an api server of the old binary version would not be able to serve a controller of the new binary version even though the emulation version is the same as the old binary version.


is not the case today? apiserver should be always be upgraded first

today although apiserver should be upgraded first according to the doc, you can still upgrade all the control plane components in one node first - apiserver1, cm1, scheduler1, before moving on to the next node. That is not a problem today because apis support N-1, N, N+1. But with emulation version, this might not work anymore.

aojea · 2025-01-20T20:05:30Z

keps/sig-architecture/4330-compatibility-versions/README.md


-Alpha APIs may not be enabled in conjunction with emulation version.
+#### Alternatives to API forward compatibility
+To make API graduation workable for controller code change under compatibility version, we have considered and rejected the following alternative options:


This also always assume that the logic in the controllers is the same between versions, but that may not be the case, if the controller and the API change between versions, or also when an existing GA API add a new field and the controller depends on that new field, you'll never being able to emulate the previous behavior ...

similar to #5038 (comment)

aojea · 2025-01-20T20:08:39Z

keps/sig-architecture/4330-compatibility-versions/README.md

+For these cases, we should try to emulate these changing features at best effort, and make sure the clients are aware of the risks of enabling default-off Beta features with emulation version.
+
+##### Alternatives
+An alternative would be to keep a exception list of `NonEmulatableFeatures`, and make the server fail to start if any of the `NonEmulatableFeatures` are enabled in compatibility mode. This approach has the benefits of predictable outcomes. The downsides are:


I think we already have this exception list, all the feature gates that are beta and disabled by default are NonEmulatableFeatures

I don't think not all the feature gates that are beta and disabled by default would break emulation necessarily, like a new field in an existing api, e.g. SELinuxMountReadWriteOncePod. But the users should be aware of the risks if they explicitly enable the disabled-by-default beta features

deads2k · 2025-01-20T20:37:45Z

keps/sig-architecture/4330-compatibility-versions/README.md

+```
+To fully emulate the controller for an older version, anywhere v1 api/type is referenced, it would need to switch to the v1beta version if the emulation version is older than the binary version. This would mean a lot of extra work, complicated testing rules, and high maintenance cost even for simple API graduations, while the emulation fidelity is unreliable with the extra complexity.
+
+So instead of truly emulating the feature controllers and API availability at the emulation version, we are proposing to keep forward compatibility of all APIs in compatibility version mode and have the non-emulatable feature controllers to update to use the newest available API. 


So if a 1.32 controller does thing/A. A 1.33 controller does thing/A and also does wrongThing/B. Does this meant that switching to an emulation version for 1.32 does not stop the 1.33 controller from doing wrongThing/B?

deads2k · 2025-01-20T20:39:21Z

keps/sig-architecture/4330-compatibility-versions/README.md

+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` | 1.30 | NA
+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` | 1.31 | `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`
+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.33` | 1.33 | `api/v1`
+`v1beta1: introduced=1.31, removed=1.32`<br>`v1beta2: introduced=1.32` | 1.31 | `api/v1beta1` and `api/v1beta2` available only when `--runtime-config=api/v1beta1=true`


why is v1beta2 on in this scenario since it wasn't even introduced in 1.31?

v1beta2 and v1beta1 have the same stability level, so the controller may have moved on to v1beta2 in the binary. By forward compatibility, we are exposing apis that are introduced later than the emulation version that are the same / more stable then the apis enabled at the emulation version.

Signed-off-by: Siyuan Zhang <[email protected]>

cbandy · 2025-01-25T03:45:27Z

keps/sig-architecture/4330-compatibility-versions/README.md

Why is this feature described as Removed: 1.31 in the 1.30 release below?

enhancements/keps/sig-architecture/4330-compatibility-versions/README.md

Lines 438 to 449 in 41a49ff

The steps to remove the Beta feature would be:

| Release | Stage | Feature tracking information |

| ------- | ----- | ------------------------------------------------- |

| 1.26 | beta | Beta: 1.26 |

| 1.27 | beta | Beta: 1.26, Deprecated: 1.27 |

| 1.28 | beta | Beta: 1.26, Deprecated: 1.27 |

| 1.29 | beta | Beta: 1.26, Deprecated: 1.27 |

| 1.30 | - | Beta: 1.26, Deprecated: 1.27, Removed: 1.31 |

| 1.31 | - | Beta: 1.26, Deprecated: 1.27, Removed: 1.31 |

| 1.32 | - | Beta: 1.26, Deprecated: 1.27, Removed: 1.31 |

| 1.33 | - | **`if featureGate enabled { // implement feature }` code may be removed at this step** |

cbandy · 2025-01-25T04:06:30Z

keps/sig-architecture/4330-compatibility-versions/README.md

+-----|-----|-----
+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` | 1.30 | NA because we do not support emulating alpha apis.
+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` | 1.31 | `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`
+`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.33` | 1.33 | `api/v1`


There appears to be a gap here: beta removed in 1.32, but v1 introduced in 1.33.

Suggested change

`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.33` | 1.33 | `api/v1`

`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` | 1.33 | `api/v1`

cbandy · 2025-01-25T04:28:18Z

keps/sig-architecture/4330-compatibility-versions/README.md

-Beta API graduated to GA|off-by-default|on|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available
-Beta API removed|off-by-default|-|API available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist
-on-by-default Beta API removed|on-by-default|-|API available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist
+Beta API graduated to GA|off-by-default|on|API `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available


Does "graduated to GA" imply that the Beta API is removed (from N)?

Should --runtime-config=api/v1beta1=true have some effect when emulation-version=N?

cbandy · 2025-01-25T04:33:31Z

keps/sig-architecture/4330-compatibility-versions/README.md

-on-by-default Beta API removed|on-by-default|-|API available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist
+Beta API graduated to GA|off-by-default|on|API `api/v1beta1` and `api/v1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1` available
+Beta API removed|off-by-default|-|API `api/v1beta1` available only when `--runtime-config=api/v1beta1=true`|API `api/v1beta1` does not exist
+on-by-default Beta API removed|on-by-default|-|API `api/v1beta1` available unless `--runtime-config=api/v1beta1=false`|API `api/v1beta1` does not exist
 API Storage version changed|v1beta1|v1|Resources stored as v1beta1|Resources stored as v1


Does the storage version depend on the compatibility version? If so, should it be mentioned here?

What happens in this case for Story 2, administrator needs to roll back?

1pm: emulation=N-1, everything is stored v1beta1

2pm: emulation=N, so some items are stored as v1

3pm: emulation=N-1, so...?

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 14, 2025

k8s-ci-robot requested review from dims and johnbelamaric January 14, 2025 00:57

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/architecture Categorizes an issue or PR as relevant to SIG Architecture. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jan 14, 2025

k8s-ci-robot requested review from Jefftree, jpbetz and liggitt January 14, 2025 01:00

siyuanfoundation force-pushed the compat-versions-api branch 2 times, most recently from 6abb021 to fbbc5f7 Compare January 14, 2025 05:37

Jefftree reviewed Jan 14, 2025

View reviewed changes

siyuanfoundation force-pushed the compat-versions-api branch from fbbc5f7 to ebd2f55 Compare January 14, 2025 18:04

jpbetz reviewed Jan 14, 2025

View reviewed changes

keps/sig-architecture/4330-compatibility-versions/README.md Outdated Show resolved Hide resolved

siyuanfoundation force-pushed the compat-versions-api branch 3 times, most recently from f8bce11 to 961b0c9 Compare January 14, 2025 22:39

siyuanfoundation changed the title ~~KEP-4330: Change API availability to forward compatibility and add UnEmulatableFeatures~~ KEP-4330: Change API availability to forward compatibility and add NonEmulatableFeatures Jan 15, 2025

siyuanfoundation force-pushed the compat-versions-api branch from 961b0c9 to 9d67760 Compare January 15, 2025 22:09

siyuanfoundation requested review from jpbetz and Jefftree January 16, 2025 16:55

jpbetz reviewed Jan 16, 2025

View reviewed changes

siyuanfoundation force-pushed the compat-versions-api branch 3 times, most recently from 69475d9 to 32e11d2 Compare January 16, 2025 19:42

siyuanfoundation changed the title ~~KEP-4330: Change API availability to forward compatibility and add NonEmulatableFeatures~~ KEP-4330: Change API availability to forward compatibility Jan 16, 2025

siyuanfoundation force-pushed the compat-versions-api branch from 32e11d2 to 0cdaece Compare January 16, 2025 20:55

k8s-ci-robot requested a review from deads2k January 16, 2025 20:59

siyuanfoundation force-pushed the compat-versions-api branch from 0cdaece to a101848 Compare January 16, 2025 23:32

aojea reviewed Jan 20, 2025

View reviewed changes

keps/sig-architecture/4330-compatibility-versions/README.md Show resolved Hide resolved

aojea reviewed Jan 20, 2025

View reviewed changes

deads2k reviewed Jan 20, 2025

View reviewed changes

Change API availability to forward compatibility.

32a06b7

Signed-off-by: Siyuan Zhang <[email protected]>

siyuanfoundation force-pushed the compat-versions-api branch from a101848 to 32a06b7 Compare January 21, 2025 18:26

siyuanfoundation mentioned this pull request Jan 21, 2025

KEP-1880 Multiple Service CIDRs: Graduate to GA kubernetes/kubernetes#128971

Merged

siyuanfoundation requested review from aojea, deads2k and pohly January 21, 2025 22:58

aaron-prindle mentioned this pull request Jan 23, 2025

chore: disable MultiCIDRServiceAllocator in compat version e2e test as it has known issue with --emulation-version kubernetes/test-infra#34206

Merged

cbandy reviewed Jan 25, 2025

View reviewed changes


		Alpha APIs may not be enabled in conjunction with emulation version.

		The forward compatibility of APIs should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`.

		removed) at: if an API is introduced after (or removed before) the emulation version, it should not be available at the emulation version. But in practice, that would make code changes of API graduation that wants to use newer APIs intractable.
		For example, if a controller is switching from Beta to GA API at the binary version, a full emulation would mean the whole controller code needs to be

	The steps to remove the Beta feature would be:

	\| Release \| Stage \| Feature tracking information \|
	\| ------- \| ----- \| ------------------------------------------------- \|
	\| 1.26 \| beta \| Beta: 1.26 \|
	\| 1.27 \| beta \| Beta: 1.26, Deprecated: 1.27 \|
	\| 1.28 \| beta \| Beta: 1.26, Deprecated: 1.27 \|
	\| 1.29 \| beta \| Beta: 1.26, Deprecated: 1.27 \|
	\| 1.30 \| - \| Beta: 1.26, Deprecated: 1.27, Removed: 1.31 \|
	\| 1.31 \| - \| Beta: 1.26, Deprecated: 1.27, Removed: 1.31 \|
	\| 1.32 \| - \| Beta: 1.26, Deprecated: 1.27, Removed: 1.31 \|
	\| 1.33 \| - \| `if featureGate enabled { // implement feature }` code may be removed at this step \|

	`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.33` \| 1.33 \| `api/v1`
	`v1alpha1: introduced=1.30, removed=1.31`<br>`v1beta1: introduced=1.31, removed=1.32`<br>`v1: introduced=1.32` \| 1.33 \| `api/v1`

KEP-4330: Change API availability to forward compatibility #5038

Are you sure you want to change the base?

KEP-4330: Change API availability to forward compatibility #5038

Conversation

siyuanfoundation commented Jan 14, 2025 • edited Loading

k8s-ci-robot commented Jan 14, 2025

siyuanfoundation commented Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jefftree Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siyuanfoundation commented Jan 16, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siyuanfoundation Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siyuanfoundation Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siyuanfoundation commented Jan 14, 2025 •

edited

Loading

siyuanfoundation commented Jan 14, 2025 •

edited

Loading

Jefftree Jan 14, 2025 •

edited

Loading

siyuanfoundation Jan 21, 2025 •

edited

Loading

siyuanfoundation Jan 21, 2025 •

edited

Loading