feat(kubernetes): support corporate upstream proxy - #2633
Merged
johntmyers merged 3 commits intoAug 14, 2026
Merged
Conversation
|
All contributors have signed the DCO ✍️ ✅ |
loveRhythm1990
marked this pull request as ready for review
August 6, 2026 13:48
loveRhythm1990
requested review from
a team,
derekwaynecarr,
maxamillion and
mrunalp
as code owners
August 6, 2026 13:48
loveRhythm1990
force-pushed
the
2624-kubernetes-corporate-proxy/lr
branch
2 times, most recently
from
August 6, 2026 13:52
26493d1 to
78250e4
Compare
johntmyers
reviewed
Aug 13, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Validation: Project-valid as a focused Kubernetes-driver extension of the corporate upstream-proxy support merged in #2245, with the Kubernetes-specific operator path and motivation documented in #2624.
Head SHA: 78250e45a3ade0622f167be52f1499e190ecc917
Base SHA: 0c7e59a95355cabc15ccaddb86fcbe6a1d30eaaa
Merge base SHA: d2c44b0e5393e3746eae5783aa99ac31be08daae
Patch ID: 5db52dea1e5bdc062384aa8edc68b640bfa32b9d
Gator payload: 4
Review mode: initial
Previous reviewed SHA: none
Review budget exhausted: no
Maintainer decision required: no
Blocking findings:
GATOR-78250e45-01: Kubernetes Secret keys longer than 253 bytes pass gateway validation even though Kubernetes cannot create them, shifting an invalid deployment setting into repeated sandbox Pod-provisioning failures.
Carried findings:
- None
Non-blocking suggestions:
- Consider adding corporate-proxy checks to
debug-openshell-clusterfor rendered configuration, Secret-volume events, supervisor arguments/mounts, and proxy reachability.
Docs: Fern documentation covers the new Kubernetes setup and gateway/compute-driver configuration; existing navigation already exposes those pages.
Next state: gator:in-review
johntmyers
added
the
gator:in-review
Collaborator
|
Thanks @loveRhythm1990 - started our review agent. I'm also soliciting some reviews from other maintainers. |
Contributor
|
I took a pass on this and wasn't able to identify any issues beyond the one already highlighted by John's agent. Nice work |
Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
…t create Gateway validation accepted proxy_auth_secret_key values that Kubernetes rejects when creating the Secret (keys longer than 253 bytes, or the reserved "."/".." names), turning an invalid deployment setting into repeated sandbox Pod-provisioning failures instead of a startup error. Reject them in validate_upstream_proxy_config so they fail closed at gateway startup. Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
…luster Add a Kubernetes corporate upstream proxy troubleshooting section covering rendered [openshell.drivers.kubernetes] configuration, credential Secret volume events, supervisor arguments and mounts confined to the network- supervising container, and proxy reachability. Signed-off-by: loveRhythm1990 <qiuweimin@126.com>
loveRhythm1990
force-pushed
the
2624-kubernetes-corporate-proxy/lr
branch
from
August 14, 2026 04:51
78250e4 to
f727a8d
Compare
Contributor
Author
|
recheck |
|
Label |
johntmyers
reviewed
Aug 14, 2026
johntmyers
left a comment
johntmyers
left a comment
Collaborator
There was a problem hiding this comment.
gator-agent
PR Review Status
Thanks @loveRhythm1990. I checked your update that rejects Kubernetes Secret keys over 253 bytes and the reserved ./.. names, including the 253-byte accepted and 254-byte rejected boundary tests. That resolves GATOR-78250e45-01. I also checked the author-only troubleshooting-skill update; it addresses the earlier non-blocking suggestion.
Validation: Project-valid as a focused Kubernetes-driver extension of the corporate upstream-proxy support, with the Kubernetes-specific operator path and motivation documented in accepted issue #2624.
Head SHA: f727a8d7128ff46cd8222482fdc1cf9893887bb7
Base SHA: c4b500a7de64d0b66e3ee8098f58d14299092162
Merge base SHA: c4b500a7de64d0b66e3ee8098f58d14299092162
Patch ID: d090894f5d8ca10c405cc12c83dc9c82038f991a
Gator payload: 4
Review mode: follow_up
Previous reviewed SHA: 78250e45a3ade0622f167be52f1499e190ecc917
Review budget exhausted: no
Maintainer decision required: no
Blocking findings:
- No blocking findings remain.
Carried findings:
GATOR-78250e45-01: resolved by the latest author delta.
Docs: Fern documentation covers the user-visible Kubernetes proxy configuration, and the debug-cluster skill now covers rendered configuration, Secret-volume events, supervisor arguments and mounts, and proxy reachability.
Next state: gator:in-review pending confirmation that the newly requested current-head E2E workflow has been queued, started, or completed.
Collaborator
|
/ok to test f727a8d |
johntmyers
added
gator:watch-pipeline
johntmyers
approved these changes
Aug 14, 2026
johntmyers
added
gator:merge-ready
and removed
gator:approval-needed
Collaborator
Monitoring CompleteMonitoring is complete because this PR has merged. Head SHA: Final status: The PR reached I removed the active |
12 tasks
johntmyers
pushed a commit
to feloy/OpenShell
that referenced
this pull request
Sep 2, 2026
…dboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com>
feloy
added a commit
to feloy/OpenShell
that referenced
this pull request
Sep 2, 2026
…dboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com>
pull Bot
pushed a commit
to rozsazoltan-forks/OpenShell
that referenced
this pull request
Sep 2, 2026
…dboxes (NVIDIA#3090) * feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): bound the proxy CA read and scope the host-loopback recipe Two review findings on the corporate forward proxy support for microVM sandboxes. The driver read the operator's proxy_ca_bundle with an unbounded fs::read and accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A special file such as /dev/zero therefore grew driver memory without bound on every authorized sandbox create, and a PEM block holding invalid DER passed the host check but contributes no trust anchor in the guest, so every supervisor would fail after boot with an error attributed to the sandbox rather than to the setting. Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it reuses the credential reader's bounded-read path (non-regular files rejected on fstat, size capped, read bounded even if the file grows), then requires at least one anchor that RootCertStore::add_parsable_certificates accepts. The supervisor's own reader now delegates to it, so host acceptance and guest acceptance are the same function and cannot drift. The published host-loopback recipe was written for libkrun only. gvproxy NATs host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run on the QEMU/TAP backend where that name resolves to the TAP host address and the driver's own nftables input chain accepts only the gateway port from the guest — no proxy on the gateway host is reachable there at any bind address, so an operator following the generic recipe lost all proxy-required egress while configuration validation succeeded. Scope the recipe to libkrun in every reference and reject a gateway-host proxy URL when a launch plan resolves to QEMU, naming the reason, instead of booting a sandbox whose policy-approved CONNECTs all time out. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): match the QEMU proxy preflight to the selected TAP host The gateway-host proxy guard added for the QEMU/TAP backend classified the wrong set of addresses in both directions. It ran at the top of configure_qemu_launch_plan, before the subnet allocation that settles plan.host_ip, so it could not compare against the address the guest actually reaches the host on. An operator pointing https_proxy at the sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the driver's nftables input chain — which accepts only the gateway port from the guest — then dropped every policy-approved CONNECT, which is exactly the silent timeout the guard exists to prevent. In the other direction it rejected 192.168.127.254 unconditionally. That address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary address that may be routable through the guest's masqueraded egress, so the guard refused a working configuration. Run the check after the launch plan's network allocation, on both the freshly-allocated and already-complete paths, and compare IP literals with that sandbox's selected TAP host. Loopback literals, localhost, and the documented host aliases that write_host_gateway_aliases seeds to the TAP host still classify as the gateway host, and the failure names the address. The gvproxy host-loopback constant returns to being a documentation anchor. Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com>
purp
pushed a commit
to purp/OpenShell
that referenced
this pull request
Sep 2, 2026
…dboxes (NVIDIA#3090) * feat(vm): support corporate HTTP forward proxy egress for microVM sandboxes The corporate forward proxy machinery from NVIDIA#1792 is driver-agnostic and already merged: openshell-supervisor-network implements CONNECT chaining, NO_PROXY matching, credentials, https:// proxies and corporate CA trust, and openshell-sandbox exposes it as six argv-only flags. Podman gained the driver half in NVIDIA#2245/NVIDIA#2512 and Kubernetes in NVIDIA#2633; the VM driver had none of it, so VM sandboxes on proxy-only networks could not reach any destination requiring the proxy even when policy allowed it. The blocking piece was not proxy logic but delivery: the VM guest init script runs as PID 1 and execs a fixed supervisor command line, and libkrun's krun_set_exec receives an empty argv, so there was no channel for driver-owned supervisor arguments. The supervisor's proxy flags deliberately have no environment fallback, and build_guest_environment merges user-supplied environment, so the guest env is not a safe transport either. Add a driver-authored argument file, mirroring the existing init.d manifest: the driver writes /opt/openshell/supervisor-args into the overlay upperdir on every launch and the guest reads it verbatim, one argument per line, appending it to every supervisor exec. It is written even when empty, which is what makes the channel unforgeable -- the upperdir always shadows the read-only image layer, so an image can neither supply its own arguments nor disable the operator's by omitting the file. Because both launch backends exec the same init script, this covers libkrun and QEMU without touching either. A microVM has no bind mounts or container secrets, so the credential and CA bundle are staged into the per-sandbox overlay the way the gateway JWT already is: credential root-only at 0600, CA at 0644, both rewritten every launch so a removed setting clears prior material, and both deleted with the sandbox state directory. This places the credential at rest in the overlay image on the gateway host, which differs from the Podman secret model and is documented as an explicit security consideration. Validation is fail-closed and shared: a new openshell_core::driver_utils::validate_upstream_proxy_settings holds the pairing rules the Podman driver established, and both the gateway and the driver call it so an invalid table names the offending key instead of surfacing as an opaque driver-readiness timeout. Guest egress leaves through gvproxy, so a proxy on the gateway host's loopback is reachable only through host.openshell.internal; the guest to gateway callback is unaffected. Closes NVIDIA#3088 Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): bound the proxy CA read and scope the host-loopback recipe Two review findings on the corporate forward proxy support for microVM sandboxes. The driver read the operator's proxy_ca_bundle with an unbounded fs::read and accepted it on a substring match for the PEM BEGIN CERTIFICATE marker. A special file such as /dev/zero therefore grew driver memory without bound on every authorized sandbox create, and a PEM block holding invalid DER passed the host check but contributes no trust anchor in the guest, so every supervisor would fail after boot with an error attributed to the sandbox rather than to the setting. Move the read into openshell-core as read_upstream_proxy_ca_bundle_file: it reuses the credential reader's bounded-read path (non-regular files rejected on fstat, size capped, read bounded even if the file grows), then requires at least one anchor that RootCertStore::add_parsable_certificates accepts. The supervisor's own reader now delegates to it, so host acceptance and guest acceptance are the same function and cannot drift. The published host-loopback recipe was written for libkrun only. gvproxy NATs host.openshell.internal to the gateway host's 127.0.0.1, but GPU sandboxes run on the QEMU/TAP backend where that name resolves to the TAP host address and the driver's own nftables input chain accepts only the gateway port from the guest — no proxy on the gateway host is reachable there at any bind address, so an operator following the generic recipe lost all proxy-required egress while configuration validation succeeded. Scope the recipe to libkrun in every reference and reject a gateway-host proxy URL when a launch plan resolves to QEMU, naming the reason, instead of booting a sandbox whose policy-approved CONNECTs all time out. Signed-off-by: Philippe Martin <phmartin@redhat.com> * fix(vm): match the QEMU proxy preflight to the selected TAP host The gateway-host proxy guard added for the QEMU/TAP backend classified the wrong set of addresses in both directions. It ran at the top of configure_qemu_launch_plan, before the subnet allocation that settles plan.host_ip, so it could not compare against the address the guest actually reaches the host on. An operator pointing https_proxy at the sandbox's own TAP host address, such as 10.0.128.1, passed the check, and the driver's nftables input chain — which accepts only the gateway port from the guest — then dropped every policy-approved CONNECT, which is exactly the silent timeout the guard exists to prevent. In the other direction it rejected 192.168.127.254 unconditionally. That address is special only to libkrun/gvproxy; on QEMU/TAP it is an ordinary address that may be routable through the guest's masqueraded egress, so the guard refused a working configuration. Run the check after the launch plan's network allocation, on both the freshly-allocated and already-complete paths, and compare IP literals with that sandbox's selected TAP host. Loopback literals, localhost, and the documented host aliases that write_host_gateway_aliases seeds to the TAP host still classify as the gateway host, and the failure names the address. The gvproxy host-loopback constant returns to being a documentation anchor. Signed-off-by: Philippe Martin <phmartin@redhat.com> --------- Signed-off-by: Philippe Martin <phmartin@redhat.com>
craig-kindo
added a commit
to UsableMachines/OpenShell
that referenced
this pull request
Sep 9, 2026
…roxy Ported from 8a55af9 and 2364978 onto upstream main. Not cherry-picked: both sat on config plumbing upstream has since landed itself (NVIDIA#2633), so a pick conflicts in 13 files while re-adding fields that already exist. Three new flags. Everything else those commits carried is now upstream's: - --upstream-proxy-auth-sandbox-identity sends the resolved sandbox id as the Proxy-Authorization Basic username with an empty password, so a proxy can attribute and ACL egress per sandbox instead of seeing one client - --upstream-proxy-client-cert / --upstream-proxy-client-key present a client certificate on the TLS hop to a proxy that authenticates its callers - --upstream-proxy-ca from the source commit is NOT added; it duplicated upstream's existing --upstream-proxy-ca-bundle, which is used instead This is what backend/sandbox in the mono repo requires, and both halves are needed: egress/identity.go parses exactly `Proxy-Authorization: Basic base64("<sandbox-id>:")` for the caller's identity, and egress/main.go sets ClientAuth: RequireAndVerifyClientCert, so the supervisor must also present a certificate. Identity comes from the header, authentication from the cert. Fail-closed, each with a test: - an unresolved sandbox identity refuses to start rather than egressing unattributed, which would silently defeat a proxy ACL keyed on the username - mutually exclusive with the auth file, because one request carries a single Proxy-Authorization header, enforced at the driver and the supervisor - it rejects rather than requires the cleartext-credential acknowledgement, whose text would be false when nothing confidential is sent - half a client identity, or either half against an http:// proxy where it would be silently ignored, is fatal at startup - the client cert requires supervisor topology "sidecar", keeping the private key mount out of the workload container Verified: cargo check --workspace --all-targets clean; 1268 supervisor-network, 253 driver-kubernetes and 461 core tests pass, 0 failed; helm lint clean and the chart renders.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Kubernetes support for the existing supervisor corporate HTTP forward-proxy path. This lets sandboxes in proxy-only clusters reach policy-approved TLS destinations through an operator-managed proxy without allowing workloads to choose, bypass, or observe the upstream route.
The Kubernetes driver previously had no configuration or Secret-projection path for the shared supervisor feature that Podman already uses. This PR adds that driver-level plumbing while preserving direct egress as the default when no proxy is configured.
Related Issue
Closes #2624
Changes
https_proxy,no_proxy, proxy credential Secret name/key, insecure Basic-auth acknowledgement, and CONNECT-by-hostname mode. The URL remains intentionally limited tohttp://forward proxies; HTTPS-to-proxy support is out of scope.upstreamProxyvalues, while keeping it deployment-owned rather than allowing per-sandboxdriver_configoverrides.topology = \"sidecar\"whenever proxy credentials are configured. Combined topology shares a container filesystem with the workload, and workspacefsGroupcan make an otherwise root-only Secret mount group-readable; rejecting this combination avoids a credential disclosure path. Credential-free proxy configuration can still use Combined topology.network-initnever receives proxy configuration and the agent container never receives the credential mount.Testing
mise run pre-commitpassesNO_PROXYbypasses only the corporate proxy; policy-denied traffic remains deniedmise run testhas one unrelated environment-dependent DNS failure:test_forward_public_ip_allowed_without_allowed_ipsresolveddns.googleto198.18.1.240, which the SSRF guard correctly rejects. All Kubernetes proxy-specific checks above passed.Checklist