|
| 1 | +# NPEP-137: Conformance Profiles |
| 2 | + |
| 3 | +* Issue: [#137](https://github.com/kubernetes-sigs/network-policy-api/issues/137) |
| 4 | +* Status: Implementable |
| 5 | + |
| 6 | +## TLDR |
| 7 | + |
| 8 | +Add conformance profiles for existing conformance tests which implementations |
| 9 | +can leverage when running the conformance test suite in their downstream repos |
| 10 | +to report the results of the tests back to the Network Policy API project and |
| 11 | +receive recognition (eg. "conformance badge") |
| 12 | + |
| 13 | +* NOTE: Adapted with love from https://gateway-api.sigs.k8s.io/geps/gep-1709/ (Then |
| 14 | +a question may arise, why do we need our own NPEP? Answer: Gateway API Project has |
| 15 | +lots of resources and features and it gets very complicated to adopt their same API. |
| 16 | +We need to tune the profiles to be compatible with what works for Network Policy API |
| 17 | +project, thus the need for this NPEP). Our profiles are much simpler than what GatewayAPI |
| 18 | +supports. |
| 19 | + |
| 20 | +## Goals |
| 21 | + |
| 22 | +* Add conformance profiles for existing tests that downstream implementations |
| 23 | +can subscribe to to run tests associated to the supported API feature sets |
| 24 | + |
| 25 | +* Add a reporting mechanism via a new CRD where conformance results can be |
| 26 | +reported back to the Network Policy API project and provide "conformance badges" |
| 27 | +via Network Policy official website to recognize these implementations. |
| 28 | + |
| 29 | +* Expand existing conformance testing framework to account for profiles |
| 30 | + |
| 31 | +## Non-Goals |
| 32 | + |
| 33 | +* The first iteration will just have a simple repo under which the report |
| 34 | +results will be stored, no automated reporting infrastructure will be built for this now. |
| 35 | +Implementations will need to open PRs against network-policy-api repo to upload |
| 36 | +the results. |
| 37 | + |
| 38 | +## Introduction |
| 39 | + |
| 40 | +Currently, the conformance tests are grouped into `CoreFeatures` and |
| 41 | +`ExtendedFeatures`. The support for features that fall under the `CoreFeatures` |
| 42 | +are a requirement for conformant implementations, while the support for features that fall |
| 43 | +under `ExtendedFeatures` are optional and do not gate API conformance for implementations. |
| 44 | + |
| 45 | +In this NPEP, we will add a concept called `named profiles` which will indicate a |
| 46 | +"level of conformance" for a given resource. We are picking profiles on a resource |
| 47 | +level because that is what makes sense for the project as of today. The initial decision |
| 48 | +to report conformance profiles on a per-resource level is a choice which may be re-evaluated |
| 49 | +by the NetworkPolicy API working group in the future. Using these `named profiles`, |
| 50 | +implementations can run the conformance tests, prove they satisfy the requirements for each profile, |
| 51 | +and receive conformance badges from the Network Policy API project. |
| 52 | + |
| 53 | +This NPEP aims to provide an API (CRD) for reporting results from these conformance |
| 54 | +tests. CLI tooling will also be provided for invoking the conformance tests with |
| 55 | +the profile names so that the results can be easily submitted back upstream. |
| 56 | + |
| 57 | +## User-Stories/Use-Cases |
| 58 | + |
| 59 | +Story 1: Receive an API conformance badge |
| 60 | + |
| 61 | +As a CNI maintainer, I want to receive recognition from Network Policy API subgroup |
| 62 | +that my plugin implements the `AdminNetworkPolicy` API. This badge will enable us |
| 63 | +to advertise upstream feature support in our plugin easily to our users. |
| 64 | + |
| 65 | +Story 2: Track Implementations for each API resource |
| 66 | + |
| 67 | +As a Network Policy project maintainer, I would like to easily track which |
| 68 | +implementations are using the defined API resources in our project so that it |
| 69 | +paves way for easier collaboration and feedback loops when changes are introduced |
| 70 | +to the API in each release. |
| 71 | + |
| 72 | +## API |
| 73 | + |
| 74 | +We will add a new API resource called `ConformanceReport` which will be at |
| 75 | +the center of our test result reporting workflow. The implementors running |
| 76 | +the tests can then: |
| 77 | + |
| 78 | +1. Choose a `named profile` |
| 79 | +2. Integrate the tests required by a given profile in their downstream project |
| 80 | +3. Report the results to Network Policy API project using this new API resource |
| 81 | +4. Get a conformance badge recognition via our official website |
| 82 | + |
| 83 | +## Profiles |
| 84 | + |
| 85 | +Named Profiles for Network Policy API project will be tied to each API resource. |
| 86 | +We will start with two named profiles and expand this as the project evolves. |
| 87 | + |
| 88 | +1. AdminNetworkPolicy |
| 89 | +2. BaselineAdminNetworkPolicy |
| 90 | + |
| 91 | +Each of these profiles may have a combination of conformance tests that fall under |
| 92 | +`CoreFeatures` and `ExtendedFeatures`. Example; if you pick the profile |
| 93 | +`AdminNetworkPolicy`, all tests like `AdminNetworkPolicyEgressSCTP` and |
| 94 | +`AdminNetworkPolicyPriorityField` fall under the `SupportAdminNetworkPolicy` feature |
| 95 | +which is under the `CoreFeatures` subset. So these tests must pass for the |
| 96 | +`AdminNetworkPolicy` profile conformance. Whereas tests (we don't have any today) |
| 97 | +that fall under `AdminNetworkPolicySameLabels` feature which is under the |
| 98 | +`ExtendedFeatures` subset are not mandatory for `AdminNetworkPolicy` |
| 99 | +profile conformance. |
| 100 | + |
| 101 | +## Integration |
| 102 | + |
| 103 | +The conformance profile test suite can be integrated, invoked and run from your implementation |
| 104 | +using two methods: |
| 105 | + |
| 106 | +* The `go test` CLI commands specifying the required information need to generate |
| 107 | +a conformance test report. Sample: |
| 108 | +``` |
| 109 | +go test -v ./conformance -run TestConformanceProfiles -args --conformance-profiles=AdminNetworkPolicy,BaselineAdminNetworkPolicy --organization=ovn-org -project=ovn-kubernetes -url=<project-url> -version=0.1.1 -contact=<> -additional-info=<link-to-implementation> |
| 110 | +``` |
| 111 | +* Using the conformance profile test suite `TestConformanceProfiles` by directly customizing it by providing |
| 112 | +the correct arguments. Sample: |
| 113 | +``` |
| 114 | +cpSuite := suite.NewConformanceProfileTestSuite( |
| 115 | + suite.ConformanceProfileOptions{ |
| 116 | + suite.Options: cSuiteDefaultOptions, |
| 117 | + Implementation: confv1alpha1.Implementation{ |
| 118 | + Organization: "ovn-org", |
| 119 | + Project: "OVN-Kubernetes CNI", |
| 120 | + URL: "https://github.com/ovn-org/ovn-kubernetes", |
| 121 | + Version: "0.1.1", |
| 122 | + Contact: []string{"@tssurya"}, |
| 123 | + AdditionalInformation: "https://github.com/ovn-org/ovn-kubernetes/blob/1c9f73dc8a755c07b22858c7404a7884970d1989/test/conformance/network_policy_v2_test.go" |
| 124 | + }, |
| 125 | + ConformanceProfiles: sets.New( |
| 126 | + suite.ANPConformanceProfileName, |
| 127 | + suite.BANPConformanceProfileName, |
| 128 | + ), |
| 129 | + }) |
| 130 | +``` |
| 131 | + |
| 132 | +## Reporting process and certification |
| 133 | + |
| 134 | +The reporting process is related to a specific API's version and channel (core and experimental). |
| 135 | +There are fields in the ConformanceReport CRD that includes such information. Any implementation |
| 136 | +can run the existing conformance test suite specifying the profiles they support and that will |
| 137 | +generate an output that looks like this: |
| 138 | + |
| 139 | +``` |
| 140 | + Conformance report: |
| 141 | + apiVersion: policy.networking.k8s.io/v1alpha1 |
| 142 | + date: "2023-10-03T08:15:25+02:00" |
| 143 | + implementation: |
| 144 | + contact: |
| 145 | + - "@tssurya" |
| 146 | + organization: ovn-org |
| 147 | + project: OVN-Kubernetes CNI |
| 148 | + url: "https://github.com/ovn-org/ovn-kubernetes" |
| 149 | + version: 0.1.1 |
| 150 | + additionalInformation: "https://github.com/ovn-org/ovn-kubernetes/blob/1c9f73dc8a755c07b22858c7404a7884970d1989/test/conformance/network_policy_v2_test.go" |
| 151 | + kind: ConformanceReport |
| 152 | + networkPolicyV2APIVersion: v0.1.1 |
| 153 | + profiles: |
| 154 | + - core: |
| 155 | + failedTests: |
| 156 | + - AdminNetworkPolicyIngressSCTP |
| 157 | + - AdminNetworkPolicyEgressUDP |
| 158 | + result: failure |
| 159 | + statistics: |
| 160 | + Failed: 2 |
| 161 | + Passed: 5 |
| 162 | + Skipped: 0 |
| 163 | + summary: "" |
| 164 | + name: AdminNetworkPolicy |
| 165 | + - core: |
| 166 | + result: success |
| 167 | + statistics: |
| 168 | + Failed: 0 |
| 169 | + Passed: 7 |
| 170 | + Skipped: 0 |
| 171 | + summary: "" |
| 172 | + name: BaselineAdminNetworkPolicy |
| 173 | +``` |
| 174 | + |
| 175 | +This can then be uploaded to network-policy-api/conformance/reports/v.x.x/cni-name.yaml by |
| 176 | +opening a PR. That will then be reviewed and approved by maintainers thus recognizing the |
| 177 | +implementations that are conformant. It is recommended for the implementations to use the |
| 178 | +`additionalInformation` field to provide links to the implementation or github actions or |
| 179 | +jenkins or any other CI/CD job definitions that helped generate this report. This will |
| 180 | +help maintainers make an informed decision on merging the report PR. |
| 181 | + |
| 182 | +## Alternatives |
| 183 | + |
| 184 | +N/A - We just went with what Gateway API project already has implemented without |
| 185 | +having to reinvent the wheel. |
| 186 | + |
| 187 | +## References |
| 188 | + |
| 189 | +1. https://github.com/kubernetes-sigs/network-policy-api/pull/142 |
0 commit comments