ai-gateway: raise drainTimeout so model-deploy churn doesn't kill live completions

[johnl-amd]

Summary

Stops long LLM completions on the ai-gateway from being cut mid-stream when the listener churns on model deploy/undeploy — without slowing pod termination.

Problem

Every model deploy/undeploy changes the ai-gateway filter set (per-model EPP ext_proc + catch-all ext_authz filters). That triggers an Envoy in-place filter-chain update: the superseded chain is drained and any connection still on it is force-closed at the end of the drain window. With the 60s default, an in-flight streaming completion is severed mid-body — measured on app-dev: an unrelated model's event killed a held ai.<domain> stream ~60s later (CURLE_PARTIAL_FILE). It lands entirely on the longest-running requests, so it's invisible in ordinary testing.

Change

Raise EnvoyProxy.shutdown.drainTimeout to 31m on the ai-gateway EnvoyProxy. Envoy Gateway feeds drainTimeout into Envoy's --drain-time-s, which governs LDS-triggered filter-chain draining on a running pod, so a churn-superseded chain now holds its connections long enough for the completion to finish and close naturally (≥ the ai-gateway-discovery REQUEST_TIMEOUT of 30m). A drained chain is released as soon as its connections close, so cost is bounded by concurrent long streams, not by churn rate.

Decoupled from pod termination (important)

EG derives terminationGracePeriodSeconds = drainTimeout + 5m, so a bare 31m would make every pod termination (rollout, node drain, autoscaler scale-down, upgrade) block up to 36m — and hit that ceiling on every rollout via the lingering-Prometheus-scrape drain bug (envoyproxy/gateway#4125). To avoid that, envoyDeployment.patch forces terminationGracePeriodSeconds back to the 360s default. The two timers are independent at runtime:

• 31m → --drain-time-s: the running-pod LDS churn drain (protects in-flight streams).
• 360s → the k8s SIGKILL deadline on termination (rollouts/node drains stay at today's 6m).

Scope / blast radius

• ai-gateway only (Gateway-scoped EnvoyProxy; the GatewayClass has no cluster-wide EnvoyProxy). The shared https gateway and everything else are untouched.
• Does not reduce churn — it stops churn from cutting live streams. The real filter-topology fix is upstream (envoyproxy/ai-gateway EPP filter dedup + generated-rule naming; #1917).
• Rollovers still cut streams older than 6m (unchanged status quo).

Test plan

• helm template … --set aiGateway.enabled=true renders drainTimeout: 31m + the terminationGracePeriodSeconds: 360 patch; helm lint clean.
• On deploy, verify the patch stuck (we override an EG-managed field):

  kubectl get deploy -n envoy-gateway-system <ai-gateway-deploy> \
    -o jsonpath='{.spec.template.spec.terminationGracePeriodSeconds}'   # expect 360, not 2160
  

• Churn check: hold a >60s stream on ai.<domain>, deploy/undeploy another model, confirm the stream is not dropped and total_filter_chains_draining (not total_listeners_draining) moved.

[tomastola]

Checked this against the EG source and against measurements I took on app-dev. The mechanism is right and the effect is verified — two notes below, neither structural.

Verified

The grace-period derivation is exactly as described. internal/infrastructure/kubernetes/proxy/resource_provider.go:614-623 — s = int(d.Seconds() + 300), default 360. So a bare 31m really would give 2160s. Good catch.

The patch will win — this is settleable from source, not just "verify on deploy". In resource_provider.go, line 422 sets TerminationGracePeriodSeconds: expectedTerminationGracePeriodSeconds(...) while building the Deployment, and line 449 then runs utils.MergeWithPatch(deployment, deploymentConfig.Patch). The patch is applied after the computed value, so the StrategicMerge override takes effect. The verification step in the test plan can drop from "confirm the override works at all" to a sanity check.

For completeness on the other end of the mapping: EG's --drain-time-s default is 60.0 (internal/infrastructure/common/proxy_args.go:87), overridden by shutdown.drainTimeout when set. So this change moves the running-pod churn drain 60s → 1860s. (Note Envoy's native default for that flag is 600s — EG doesn't use it, so the 10-minute figure from Envoy's docs never applies here.)

The effect is measured, not theoretical. While the 31m was live on app-dev I re-ran the exact experiment that first demonstrated the problem: held a stream on ai.<domain>, then deleted an unrelated model's -catchall SecurityPolicy. Churn was identical to the failing run — listener_in_place_updated 3→5, total_filter_chains_draining 3→5 — and the stream survived 208s and was still going. Under the 60s default the same trigger killed it at 61s with CURLE_PARTIAL_FILE. Same churn, only the drain window changed. Worth putting in the PR description; it isolates the drain window as the sole variable.

Findings

1. The #4125 citation doesn't hold. envoyproxy/gateway#4125 is closed as COMPLETED, 2024-09-21 — roughly two years before v1.8.1. Presenting it as a live failure mode ("hit that ceiling on every rollout via the lingering-Prometheus-scrape drain bug") isn't accurate for the version we run.

The decoupling still stands on its own merits — 36 minutes of termination grace on every rollout, node drain and autoscaler scale-down is bad regardless of whether anything forces it to the ceiling — so I'd correct the justification rather than change the code.

2. Worth confirming the 6m tradeoff is the intended one. With terminationGracePeriodSeconds pinned at 360, a pod termination gives in-flight work 6 minutes before SIGKILL, regardless of the 31m churn drain. The PR says this plainly, which is good. The consequence worth being explicit about: once this lands, gateway rollout becomes the dominant remaining cause of severed completions — a 30m completion still dies on any proxy restart, upgrade or node drain. That seems like the right trade (churn is frequent and unpredictable; rollouts are planned and rare), but it's the thing to watch afterwards rather than assume is solved.

3. Nice to know, not blocking. The https gateway carries workloads.<domain> as well as the apps, and I measured a stream there hard-closed at 60s by the same mechanism. #4376 should make that listener's filter set static with respect to model activity, so leaving https out is reasonable — just flagging that the exclusion is load-bearing on #4376 landing, and that other sources of https listener churn would still cut those streams at 60s.

4. Minor. "Cost is bounded by concurrent long streams, not by churn rate" is the right framing — better than the "chains pile up per churn event" worry I had earlier. Chains do accumulate while their connections live (I saw 3, then 5, draining concurrently on app-dev), but they're released as soon as those close, so the bound holds.

[woojae-siloai]

LGTM