Skip to content

Commit ace59d6

Browse files
openshift-merge-bot[bot]openshift-merge-bot[bot]
authored
Merge pull request #2186 from alebedev87/4-17-idle-conn-termination-policy
[release-4.17] OCPBUGS-49702: Add IdleConnectionTerminationPolicy field to IngressControllerSpec
2 parents 0cfc958 + 40536da commit ace59d6

File tree

11 files changed

+304
-19
lines changed

11 files changed

+304
-19
lines changed

‎openapi/generated_openapi/zz_generated.openapi.go

Lines changed: 8 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎openapi/openapi.json

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28237,6 +28237,11 @@
2823728237
"description": "httpHeaders defines policy for HTTP headers.\n\nIf this field is empty, the default values are used.",
2823828238
"$ref": "#/definitions/com.github.openshift.api.operator.v1.IngressControllerHTTPHeaders"
2823928239
},
28240+
"idleConnectionTerminationPolicy": {
28241+
"description": "idleConnectionTerminationPolicy maps directly to HAProxy's idle-close-on-response option and controls whether HAProxy keeps idle frontend connections open during a soft stop (router reload).\n\nAllowed values for this field are \"Immediate\" and \"Deferred\". The default value is \"Deferred\".\n\nWhen set to \"Immediate\", idle connections are closed immediately during router reloads. This ensures immediate propagation of route changes but may impact clients sensitive to connection resets.\n\nWhen set to \"Deferred\", HAProxy will maintain idle connections during a soft reload instead of closing them immediately. These connections remain open until any of the following occurs:\n\n - A new request is received on the connection, in which\n case HAProxy handles it in the old process and closes\n the connection after sending the response.\n\n - HAProxy's `timeout http-keep-alive` duration expires\n (300 seconds in OpenShift's configuration, not\n configurable).\n\n - The client's keep-alive timeout expires, causing the\n client to close the connection.\n\nSetting Deferred can help prevent errors in clients or load balancers that do not properly handle connection resets. Additionally, this option allows you to retain the pre-2.4 HAProxy behaviour: in HAProxy version 2.2 (OpenShift versions < 4.14), maintaining idle connections during a soft reload was the default behaviour, but starting with HAProxy 2.4, the default changed to closing idle connections immediately.\n\nImportant Consideration:\n\n - Using Deferred will result in temporary inconsistencies\n for the first request on each persistent connection\n after a route update and router reload. This request\n will be processed by the old HAProxy process using its\n old configuration. Subsequent requests will use the\n updated configuration.\n\nOperational Considerations:\n\n - Keeping idle connections open during reloads may lead\n to an accumulation of old HAProxy processes if\n connections remain idle for extended periods,\n especially in environments where frequent reloads\n occur.\n\n - Consider monitoring the number of HAProxy processes in\n the router pods when Deferred is set.\n\n - You may need to enable or adjust the\n `ingress.operator.openshift.io/hard-stop-after`\n duration (configured via an annotation on the\n IngressController resource) in environments with\n frequent reloads to prevent resource exhaustion.",
28242+
"type": "string",
28243+
"default": "Deferred"
28244+
},
2824028245
"logging": {
2824128246
"description": "logging defines parameters for what should be logged where. If this field is empty, operational logs are enabled but access logs are disabled.",
2824228247
"$ref": "#/definitions/com.github.openshift.api.operator.v1.IngressControllerLogging"

‎operator/v1/tests/ingresscontrollers.operator.openshift.io/AAA_ungated.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ tests:
1414
kind: IngressController
1515
spec:
1616
httpEmptyRequestsPolicy: Respond
17+
idleConnectionTerminationPolicy: Deferred
1718
- name: Should be able to create an IngressController with valid Actions
1819
initial: |
1920
apiVersion: operator.openshift.io/v1
@@ -74,6 +75,7 @@ tests:
7475
kind: IngressController
7576
spec:
7677
httpEmptyRequestsPolicy: Respond
78+
idleConnectionTerminationPolicy: Deferred
7779
httpHeaders:
7880
actions:
7981
response:
@@ -495,6 +497,7 @@ tests:
495497
namespace: openshift-ingress-operator
496498
spec:
497499
httpEmptyRequestsPolicy: Respond
500+
idleConnectionTerminationPolicy: Deferred
498501
tuningOptions:
499502
connectTimeout: 10s
500503
- name: Should be able to create an IngressController with valid composite connect timeout
@@ -515,6 +518,7 @@ tests:
515518
namespace: openshift-ingress-operator
516519
spec:
517520
httpEmptyRequestsPolicy: Respond
521+
idleConnectionTerminationPolicy: Deferred
518522
tuningOptions:
519523
connectTimeout: 100ms300μs
520524
- name: Should be able to create an IngressController with valid fraction connect timeout
@@ -535,6 +539,7 @@ tests:
535539
namespace: openshift-ingress-operator
536540
spec:
537541
httpEmptyRequestsPolicy: Respond
542+
idleConnectionTerminationPolicy: Deferred
538543
tuningOptions:
539544
connectTimeout: 1.5m
540545
- name: Should not be able to create an IngressController with invalid unit connect timeout

‎operator/v1/tests/ingresscontrollers.operator.openshift.io/IngressControllerLBSubnetsAWS.yaml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ tests:
2929
kind: IngressController
3030
spec:
3131
httpEmptyRequestsPolicy: Respond
32+
idleConnectionTerminationPolicy: Deferred
3233
endpointPublishingStrategy:
3334
type: LoadBalancerService
3435
loadBalancer:
@@ -70,6 +71,7 @@ tests:
7071
kind: IngressController
7172
spec:
7273
httpEmptyRequestsPolicy: Respond
74+
idleConnectionTerminationPolicy: Deferred
7375
endpointPublishingStrategy:
7476
type: LoadBalancerService
7577
loadBalancer:
@@ -183,6 +185,7 @@ tests:
183185
kind: IngressController
184186
spec:
185187
httpEmptyRequestsPolicy: Respond
188+
idleConnectionTerminationPolicy: Deferred
186189
endpointPublishingStrategy:
187190
type: LoadBalancerService
188191
loadBalancer:
@@ -473,6 +476,7 @@ tests:
473476
kind: IngressController
474477
spec:
475478
httpEmptyRequestsPolicy: Respond
479+
idleConnectionTerminationPolicy: Deferred
476480
endpointPublishingStrategy:
477481
type: LoadBalancerService
478482
loadBalancer:
@@ -530,6 +534,7 @@ tests:
530534
kind: IngressController
531535
spec:
532536
httpEmptyRequestsPolicy: Respond
537+
idleConnectionTerminationPolicy: Deferred
533538
endpointPublishingStrategy:
534539
type: LoadBalancerService
535540
loadBalancer:
@@ -579,6 +584,7 @@ tests:
579584
kind: IngressController
580585
spec:
581586
httpEmptyRequestsPolicy: Respond
587+
idleConnectionTerminationPolicy: Deferred
582588
endpointPublishingStrategy:
583589
type: LoadBalancerService
584590
loadBalancer:

‎operator/v1/tests/ingresscontrollers.operator.openshift.io/SetEIPForNLBIngressController.yaml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ tests:
1414
kind: IngressController
1515
spec:
1616
httpEmptyRequestsPolicy: Respond
17+
idleConnectionTerminationPolicy: Deferred
1718
- name: Should allow to set NLB parameters when LBType is NLB.
1819
initial: |
1920
apiVersion: operator.openshift.io/v1
@@ -42,6 +43,7 @@ tests:
4243
namespace: openshift-ingress-operator
4344
spec:
4445
httpEmptyRequestsPolicy: Respond
46+
idleConnectionTerminationPolicy: Deferred
4547
endpointPublishingStrategy:
4648
loadBalancer:
4749
dnsManagementPolicy: Managed
@@ -378,6 +380,7 @@ tests:
378380
namespace: openshift-ingress-operator
379381
spec:
380382
httpEmptyRequestsPolicy: Respond
383+
idleConnectionTerminationPolicy: Deferred
381384
endpointPublishingStrategy:
382385
loadBalancer:
383386
dnsManagementPolicy: Managed
@@ -432,6 +435,7 @@ tests:
432435
namespace: openshift-ingress-operator
433436
spec:
434437
httpEmptyRequestsPolicy: Respond
438+
idleConnectionTerminationPolicy: Deferred
435439
endpointPublishingStrategy:
436440
loadBalancer:
437441
dnsManagementPolicy: Managed
@@ -485,6 +489,7 @@ tests:
485489
namespace: openshift-ingress-operator
486490
spec:
487491
httpEmptyRequestsPolicy: Respond
492+
idleConnectionTerminationPolicy: Deferred
488493
endpointPublishingStrategy:
489494
loadBalancer:
490495
dnsManagementPolicy: Managed
@@ -533,6 +538,7 @@ tests:
533538
namespace: openshift-ingress-operator
534539
spec:
535540
httpEmptyRequestsPolicy: Respond
541+
idleConnectionTerminationPolicy: Deferred
536542
endpointPublishingStrategy:
537543
loadBalancer:
538544
dnsManagementPolicy: Managed
@@ -554,6 +560,7 @@ tests:
554560
namespace: openshift-ingress-operator
555561
spec:
556562
httpEmptyRequestsPolicy: Respond
563+
idleConnectionTerminationPolicy: Deferred
557564
endpointPublishingStrategy:
558565
loadBalancer:
559566
dnsManagementPolicy: Managed
@@ -566,4 +573,4 @@ tests:
566573
eipAllocations:
567574
- eipalloc-1234567890abcdefa
568575
- eipalloc-1234567890abcdefb
569-
type: LoadBalancerService
576+
type: LoadBalancerService

‎operator/v1/types_ingress.go

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -258,6 +258,75 @@ type IngressControllerSpec struct {
258258
//
259259
// +optional
260260
HTTPCompression HTTPCompressionPolicy `json:"httpCompression,omitempty"`
261+
262+
// idleConnectionTerminationPolicy maps directly to HAProxy's
263+
// idle-close-on-response option and controls whether HAProxy
264+
// keeps idle frontend connections open during a soft stop
265+
// (router reload).
266+
//
267+
// Allowed values for this field are "Immediate" and
268+
// "Deferred". The default value is "Deferred".
269+
//
270+
// When set to "Immediate", idle connections are closed
271+
// immediately during router reloads. This ensures immediate
272+
// propagation of route changes but may impact clients
273+
// sensitive to connection resets.
274+
//
275+
// When set to "Deferred", HAProxy will maintain idle
276+
// connections during a soft reload instead of closing them
277+
// immediately. These connections remain open until any of the
278+
// following occurs:
279+
//
280+
// - A new request is received on the connection, in which
281+
// case HAProxy handles it in the old process and closes
282+
// the connection after sending the response.
283+
//
284+
// - HAProxy's `timeout http-keep-alive` duration expires
285+
// (300 seconds in OpenShift's configuration, not
286+
// configurable).
287+
//
288+
// - The client's keep-alive timeout expires, causing the
289+
// client to close the connection.
290+
//
291+
// Setting Deferred can help prevent errors in clients or load
292+
// balancers that do not properly handle connection resets.
293+
// Additionally, this option allows you to retain the pre-2.4
294+
// HAProxy behaviour: in HAProxy version 2.2 (OpenShift
295+
// versions < 4.14), maintaining idle connections during a
296+
// soft reload was the default behaviour, but starting with
297+
// HAProxy 2.4, the default changed to closing idle
298+
// connections immediately.
299+
//
300+
// Important Consideration:
301+
//
302+
// - Using Deferred will result in temporary inconsistencies
303+
// for the first request on each persistent connection
304+
// after a route update and router reload. This request
305+
// will be processed by the old HAProxy process using its
306+
// old configuration. Subsequent requests will use the
307+
// updated configuration.
308+
//
309+
// Operational Considerations:
310+
//
311+
// - Keeping idle connections open during reloads may lead
312+
// to an accumulation of old HAProxy processes if
313+
// connections remain idle for extended periods,
314+
// especially in environments where frequent reloads
315+
// occur.
316+
//
317+
// - Consider monitoring the number of HAProxy processes in
318+
// the router pods when Deferred is set.
319+
//
320+
// - You may need to enable or adjust the
321+
// `ingress.operator.openshift.io/hard-stop-after`
322+
// duration (configured via an annotation on the
323+
// IngressController resource) in environments with
324+
// frequent reloads to prevent resource exhaustion.
325+
//
326+
// +optional
327+
// +kubebuilder:default:="Deferred"
328+
// +default="Deferred"
329+
IdleConnectionTerminationPolicy IngressControllerConnectionTerminationPolicy `json:"idleConnectionTerminationPolicy,omitempty"`
261330
}
262331

263332
// httpCompressionPolicy turns on compression for the specified MIME types.
@@ -2030,3 +2099,23 @@ type IngressControllerList struct {
20302099

20312100
Items []IngressController `json:"items"`
20322101
}
2102+
2103+
// IngressControllerConnectionTerminationPolicy defines the behaviour
2104+
// for handling idle connections during a soft reload of the router.
2105+
//
2106+
// +kubebuilder:validation:Enum=Immediate;Deferred
2107+
type IngressControllerConnectionTerminationPolicy string
2108+
2109+
const (
2110+
// IngressControllerConnectionTerminationPolicyImmediate specifies
2111+
// that idle connections should be closed immediately during a
2112+
// router reload.
2113+
IngressControllerConnectionTerminationPolicyImmediate IngressControllerConnectionTerminationPolicy = "Immediate"
2114+
2115+
// IngressControllerConnectionTerminationPolicyDeferred
2116+
// specifies that idle connections should remain open until a
2117+
// terminating event, such as a new request, the expiration of
2118+
// the proxy keep-alive timeout, or the client closing the
2119+
// connection.
2120+
IngressControllerConnectionTerminationPolicyDeferred IngressControllerConnectionTerminationPolicy = "Deferred"
2121+
)

‎operator/v1/zz_generated.crd-manifests/0000_50_ingress_00_ingresscontrollers.crd.yaml

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1126,6 +1126,47 @@ spec:
11261126
type: string
11271127
type: object
11281128
type: object
1129+
idleConnectionTerminationPolicy:
1130+
default: Deferred
1131+
description: "idleConnectionTerminationPolicy maps directly to HAProxy's
1132+
idle-close-on-response option and controls whether HAProxy keeps
1133+
idle frontend connections open during a soft stop (router reload).
1134+
\n Allowed values for this field are \"Immediate\" and \"Deferred\".
1135+
The default value is \"Deferred\". \n When set to \"Immediate\",
1136+
idle connections are closed immediately during router reloads. This
1137+
ensures immediate propagation of route changes but may impact clients
1138+
sensitive to connection resets. \n When set to \"Deferred\", HAProxy
1139+
will maintain idle connections during a soft reload instead of closing
1140+
them immediately. These connections remain open until any of the
1141+
following occurs: \n - A new request is received on the connection,
1142+
in which case HAProxy handles it in the old process and closes the
1143+
connection after sending the response. \n - HAProxy's `timeout http-keep-alive`
1144+
duration expires (300 seconds in OpenShift's configuration, not
1145+
configurable). \n - The client's keep-alive timeout expires, causing
1146+
the client to close the connection. \n Setting Deferred can help
1147+
prevent errors in clients or load balancers that do not properly
1148+
handle connection resets. Additionally, this option allows you to
1149+
retain the pre-2.4 HAProxy behaviour: in HAProxy version 2.2 (OpenShift
1150+
versions < 4.14), maintaining idle connections during a soft reload
1151+
was the default behaviour, but starting with HAProxy 2.4, the default
1152+
changed to closing idle connections immediately. \n Important Consideration:
1153+
\n - Using Deferred will result in temporary inconsistencies for
1154+
the first request on each persistent connection after a route update
1155+
and router reload. This request will be processed by the old HAProxy
1156+
process using its old configuration. Subsequent requests will use
1157+
the updated configuration. \n Operational Considerations: \n - Keeping
1158+
idle connections open during reloads may lead to an accumulation
1159+
of old HAProxy processes if connections remain idle for extended
1160+
periods, especially in environments where frequent reloads occur.
1161+
\n - Consider monitoring the number of HAProxy processes in the
1162+
router pods when Deferred is set. \n - You may need to enable or
1163+
adjust the `ingress.operator.openshift.io/hard-stop-after` duration
1164+
(configured via an annotation on the IngressController resource)
1165+
in environments with frequent reloads to prevent resource exhaustion."
1166+
enum:
1167+
- Immediate
1168+
- Deferred
1169+
type: string
11291170
logging:
11301171
description: logging defines parameters for what should be logged
11311172
where. If this field is empty, operational logs are enabled but

0 commit comments

Comments
 (0)