Skip to content

Commit 65de3f1

Browse files
tssuryatssurya
committed
NPEP: Iron out Egress Support API Design
Signed-off-by: Surya Seetharaman <[email protected]>
1 parent 639f674 commit 65de3f1

File tree

1 file changed

+246
-4
lines changed

1 file changed

+246
-4
lines changed

‎npep/npep-126-egress-traffic-control.md

+246-4
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# NPEP-126: Add northbound traffic support in (B)ANP API
22

33
* Issue: [#126](https://github.com/kubernetes-sigs/network-policy-api/issues/126)
4-
* Status: Provisional
4+
* Status: Implementable
55

66
## TLDR
77

@@ -76,13 +76,255 @@ selected cluster workloads to k8s-apiservers for securing the server.
7676

7777
## API
7878

79-
(... details, can point to PR with changes)
79+
Proof of Concept for the API design details can be found here: https://github.com/kubernetes-sigs/network-policy-api/pull/143
8080

81+
### Implementing egress traffic control towards cluster nodes
82+
83+
This NPEP proposes to add a new type of `AdminNetworkPolicyPeer` called `Nodes`
84+
to be able to explicitly select nodes (based on the node's labels) in the cluster.
85+
This ensures that if the list of IPs on a node OR list of nodes change, the users
86+
don't need to manually intervene to include those new IPs. The label selectors will
87+
take care of this automatically.
88+
89+
```
90+
// AdminNetworkPolicyPeer defines an in-cluster peer to allow traffic to/from.
91+
// Exactly one of the selector pointers must be set for a given peer. If a
92+
// consumer observes none of its fields are set, they must assume an unknown
93+
// option has been specified and fail closed.
94+
// +kubebuilder:validation:MaxProperties=1
95+
// +kubebuilder:validation:MinProperties=1
96+
type AdminNetworkPolicyPeer struct {
97+
<snipped>
98+
// Nodes defines a way to select a set of nodes in
99+
// in the cluster. This field follows standard label selector
100+
// semantics; if present but empty, it selects all Nodes.
101+
//
102+
// Support: Core
103+
//
104+
// +optional
105+
Nodes *metav1.LabelSelector `json:"nodes,omitempty"`
106+
}
107+
```
108+
109+
In order to ensure this type of peer cannot be set from the ingress
110+
rules, we will add the following validation rule:
111+
```
112+
+kubebuilder:validation:XValidation:rule="self.all(value, !has(value.nodes))",message="cluster-ingress traffic controls are unsupported"
113+
```
114+
This ensures nodes can be referred to only as "egress peers".
115+
116+
Example: Admin wants to deny egress traffic from tenants who don't have
117+
`restricted`, `confidential` or `internal` level security clearance
118+
to control-plane nodes at 443 and 6443 ports in the cluster
119+
120+
```
121+
apiVersion: policy.networking.k8s.io/v1alpha1
122+
kind: AdminNetworkPolicy
123+
metadata:
124+
name: node-as-egress-peers
125+
spec:
126+
priority: 55
127+
subject:
128+
namespaces:
129+
matchExpressions:
130+
- {key: security, operator: notIn, values: [restricted, confidential, internal]}
131+
egress:
132+
- name: "deny-all-egress-to-kapi-server"
133+
action: "Deny"
134+
to:
135+
- nodes:
136+
matchLabels:
137+
node-role.kubernetes.io/control-plane: true
138+
ports:
139+
- portNumber:
140+
protocol: TCP
141+
port: 443
142+
- portNumber:
143+
protocol: TCP
144+
port: 6443
145+
```
146+
147+
### Implementing egress traffic control towards external destinations
148+
149+
This NPEP proposes to add a new type of `AdminNetworkPolicyPeer` called `ExternalNetworks`
150+
to be able to explicitly select external destinations (based on the `externalNetworkSet`'s
151+
labels) in the cluster.
152+
153+
```
154+
// AdminNetworkPolicyPeer defines an in-cluster peer to allow traffic to/from.
155+
// Exactly one of the selector pointers must be set for a given peer. If a
156+
// consumer observes none of its fields are set, they must assume an unknown
157+
// option has been specified and fail closed.
158+
// +kubebuilder:validation:MaxProperties=1
159+
// +kubebuilder:validation:MinProperties=1
160+
type AdminNetworkPolicyPeer struct {
161+
<snipped>
162+
// ExternalNetworks defines a way to select ExternalNetworkSets
163+
// that consist of network CIDRs that live outside the cluster as a peer.
164+
// This field follows standard label selector semantics; if present
165+
// but empty, it selects all ExternalNetworkSets defined in the cluster.
166+
//
167+
// Support: Core
168+
//
169+
// +optional
170+
ExternalNetworks *metav1.LabelSelector `json:"externalNetworks,omitempty"`
171+
}
172+
```
173+
174+
An `externalNetworkSet` is a new object used to define a set of networks outside
175+
the cluster. This is defined as a new object so that its easy for users to group
176+
their external entities and then refer to that group from the different egress rules.
177+
This ensures consistency without the users having to manually change the ANP/BANP
178+
to accommodate the changing needs of the network cidrs. They can directly edit the
179+
new CRD which guarantees all rules effecting that peer will take effect automatically.
180+
In the future when we define `DeveloperNetworkPolicies`, this will come even more handy
181+
for uniformity specially across `Pass` actions.
182+
183+
```
184+
// ExternalNetworkSet is a cluster level resource that is used to define
185+
// a set of networks outsides the cluster which can be referred to from
186+
// the AdminNetworkPolicy && BaselineAdminNetworkPolicy APIs as an external peer
187+
type ExternalNetworkSet struct {
188+
metav1.TypeMeta `json:",inline"`
189+
metav1.ObjectMeta `json:"metadata"`
190+
191+
// Specification of the desired behavior of ExternalNetworkSet.
192+
Spec ExternalNetworkSetSpec `json:"spec"`
193+
}
194+
195+
// ExternalNetworkSetSpec defines the desired state of ExternalNetworkSet.
196+
type ExternalNetworkSetSpec struct {
197+
// Networks is the list of NetworkCIDR (both v4 & v6) that can be used to define
198+
// external destinations.
199+
// A total of 100 CIDRs will be allowed in each NetworkSet instance.
200+
// ANP & BANP APIs may use the .spec.in(e)gress.from(to).externalNetworks selector
201+
// to select a set of external networks
202+
// +optional
203+
// +kubebuilder:validation:MaxItems=100
204+
Networks []string `json:"networks,omitempty" validate:"omitempty,dive,cidr"`
205+
}
206+
```
207+
208+
In order to ensure this type of peer cannot be set from the ingress
209+
rules, we will add the following validation rule:
210+
```
211+
// +kubebuilder:validation:XValidation:rule="self.all(value, !has(value.externalNetworks))",message="cluster-ingress traffic controls are unsupported"
212+
```
213+
This ensures nodes can be referred to only as "egress peers".
214+
215+
Example: Let's define some sample sets of externalNetworks.
216+
217+
```
218+
apiVersion: policy.networking.k8s.io/v1alpha1
219+
kind: ExternalNetworkSet
220+
metadata:
221+
name: api-vm-network
222+
labels:
223+
network: intranet
224+
spec:
225+
networks:
226+
- 10.0.0.10/32
227+
- 10.0.0.11/32
228+
- 10.0.0.12/32
229+
---
230+
apiVersion: policy.networking.k8s.io/v1alpha1
231+
kind: ExternalNetworkSet
232+
metadata:
233+
name: storage-network
234+
labels:
235+
network: intranet
236+
spec:
237+
networks:
238+
- 95.90.241.22/28
239+
- 200.0.65.0/24
240+
---
241+
apiVersion: policy.networking.k8s.io/v1alpha1
242+
kind: ExternalNetworkSet
243+
metadata:
244+
name: dns-network
245+
labels:
246+
network: dns
247+
spec:
248+
networks:
249+
- 195.90.241.22/28
250+
- 111.0.65.0/29
251+
---
252+
apiVersion: policy.networking.k8s.io/v1alpha1
253+
kind: ExternalNetworkSet
254+
metadata:
255+
name: global-network
256+
labels:
257+
network: internet
258+
spec:
259+
networks:
260+
- 0.0.0.0/0
261+
```
262+
```
263+
$ oc get ens
264+
NAME NETWORKS AGE
265+
api-vm-network ["10.0.0.10/32","10.0.0.11/32","10.0.0.12/32"] 2s
266+
dns-network ["195.90.241.22/28","111.0.65.0/29"] 2s
267+
global-network ["0.0.0.0/0"] 2s
268+
storage-network ["95.90.241.22/28","200.0.65.0/24"] 2s
269+
```
270+
271+
Let's define ANP and BANP that refer to these external networks:
272+
```
273+
apiVersion: policy.networking.k8s.io/v1alpha1
274+
kind: AdminNetworkPolicy
275+
metadata:
276+
name: cluster-egress-controls
277+
spec:
278+
priority: 70
279+
subject:
280+
namespaces: {}
281+
egress:
282+
- name: "allow-all-egress-to-intranet"
283+
action: "Allow"
284+
to:
285+
- externalNetworks:
286+
matchLabels:
287+
network: intranet
288+
- name: "allow-egress-to-external-dns"
289+
action: "Allow"
290+
to:
291+
- externalNetworks:
292+
matchLabels:
293+
network: dns
294+
ports:
295+
- portNumber:
296+
protocol: UDP
297+
port: 53
298+
- name: "pass-egress-to-internet"
299+
action: "Pass"
300+
to:
301+
- externalNetworks:
302+
matchLabels:
303+
network: internet
304+
---
305+
apiVersion: policy.networking.k8s.io/v1alpha1
306+
kind: BaselineAdminNetworkPolicy
307+
metadata:
308+
name: default
309+
spec:
310+
subject:
311+
namespaces: {}
312+
egress:
313+
- name: "deny-egress-to-internet"
314+
action: "Deny"
315+
to:
316+
- externalNetworks:
317+
matchLabels:
318+
network: internet
319+
```
320+
This allows admins to specify all cluster workloads can talk to
321+
the company's internet CIDR's and the external DNS network. It
322+
also allows them to put up default guardrails of not allowing
323+
any other egress traffic towards internet in a BANP.
81324

82325
## Alternatives
83326

84-
(List other design alternatives and why we did not go in that
85-
direction)
327+
N/A
86328

87329
## References
88330

0 commit comments

Comments
 (0)