Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ Before debugging the compute platform, inspect gateway logs for failures in depe
For out-of-tree compute drivers, confirm the custom driver name and socket agree across CLI flags or `gateway.toml`, and that the operator-owned driver is running before the gateway starts:

```bash
rg -n 'compute_drivers|socket_path' /etc/openshell/gateway.toml
rg -n 'compute_driver|compute_drivers|socket_path' /etc/openshell/gateway.toml
stat /run/openshell/<driver>.sock
journalctl -u <driver-service> --no-pager --lines=200
journalctl -u openshell-gateway --no-pager --lines=200
Expand Down
2 changes: 1 addition & 1 deletion architecture/compute-runtimes.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API and CDI GPU devices when available. Delivers the supervisor via OCI image volume by default; falls back to extracting the binary to a host-side cache and bind-mounting it when `userns` is configured (overlay does not support idmapped mounts). |
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_driver = "<name>"` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |

Per-sandbox CPU and memory values currently enter the driver layer through
template resource limits. Docker and Podman apply them as runtime limits.
Expand Down
25 changes: 15 additions & 10 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -635,22 +635,27 @@ Gateway CLI flag > gateway OPENSHELL_* env var > TOML file > built-in defa
```

The TOML file is opt-in via `--config <PATH>` / `OPENSHELL_GATEWAY_CONFIG`.
Driver implementation settings live in the TOML driver tables. See
`docs/reference/gateway-config.mdx` for worked per-driver examples and RFC
0003 for the full schema.
Driver implementation settings live in the TOML driver tables. The canonical
selector is the singular `[openshell.gateway] compute_driver`; the legacy
`compute_drivers` list remains accepted and normalizes into the existing
exactly-one-driver runtime validation. See `docs/reference/gateway-config.mdx`
for worked per-driver examples and RFC 0003 for the full schema.

`database_url` is env-only and rejected when present in the file
(`OPENSHELL_DB_URL` / `--db-url`).

### Driver inheritance

`[openshell.gateway]` carries a small set of values (`sandbox_namespace`,
`default_image`,
`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`,
`host_gateway_ip`, `enable_user_namespaces`) that are inherited into each
driver's `[openshell.drivers.<name>]` table when the driver-specific table
does not override them. The allowlist is per-driver so a gateway-wide
default cannot land in a driver that does not understand it (e.g.
`[openshell.gateway]` carries shared defaults such as `default_image`,
`supervisor_image`, `guest_tls_ca/cert/key`, `client_tls_secret_name`, and
`host_gateway_ip`. It also continues to accept the historical
`sandbox_namespace`, `service_account_name`, and `enable_user_namespaces`
locations as compatibility inputs. Canonical Kubernetes configuration places
those values in `[openshell.drivers.kubernetes]` as `namespace`,
`service_account_name`, and `enable_user_namespaces`; canonical Docker
configuration uses `sandbox_label`. Driver-table values take precedence over
compatibility inputs. The allowlist is per-driver so a gateway-wide default
cannot land in a driver that does not understand it (for example,
`client_tls_secret_name` is K8s-only).

`image_pull_policy` is intentionally **not** inheritable: Kubernetes uses
Expand Down
43 changes: 41 additions & 2 deletions crates/openshell-core/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -790,11 +790,31 @@ pub struct GatewayJwtConfig {
#[serde(default = "default_gateway_id")]
pub gateway_id: String,
/// Token lifetime in seconds. A value of 0 disables expiration and is
/// intended only for local single-player deployments.
#[serde(default = "default_sandbox_token_ttl_secs")]
/// intended only for local single-player deployments. Canonical serialized
/// configuration omits the field for that non-expiring behavior; explicit
/// legacy zero remains accepted.
#[serde(
default = "default_sandbox_token_ttl_secs",
skip_serializing_if = "is_default"
)]
pub ttl_secs: u64,
}

impl GatewayJwtConfig {
/// Effective token lifetime. `None` preserves the established non-expiring
/// behavior represented by an omitted or explicit zero `ttl_secs` value.
pub fn sandbox_token_ttl(&self) -> Option<Duration> {
(self.ttl_secs != 0).then(|| Duration::from_secs(self.ttl_secs))
}
}

fn is_default<T>(value: &T) -> bool
where
T: Default + PartialEq,
{
value == &T::default()
}

fn default_gateway_id() -> String {
"openshell".to_string()
}
Expand Down Expand Up @@ -1217,6 +1237,25 @@ mod tests {
.expect("gateway JWT config should deserialize with default ttl");

assert_eq!(cfg.ttl_secs, 0);
assert_eq!(cfg.sandbox_token_ttl(), None);

let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes");
assert!(serialized.get("ttl_secs").is_none());
}

#[test]
fn gateway_jwt_positive_ttl_serializes_and_has_effective_duration() {
let cfg: GatewayJwtConfig = serde_json::from_value(serde_json::json!({
"signing_key_path": "/tmp/signing.pem",
"public_key_path": "/tmp/public.pem",
"kid_path": "/tmp/kid",
"ttl_secs": 3600
}))
.expect("gateway JWT config should deserialize with positive ttl");

assert_eq!(cfg.sandbox_token_ttl(), Some(Duration::from_secs(3600)));
let serialized = serde_json::to_value(&cfg).expect("gateway JWT config serializes");
assert_eq!(serialized["ttl_secs"], 3600);
}

#[test]
Expand Down
33 changes: 17 additions & 16 deletions crates/openshell-driver-docker/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,9 @@ pub struct DockerComputeConfig {
/// Image pull policy for sandbox images.
pub image_pull_policy: String,

/// Namespace label applied to Docker sandboxes.
pub sandbox_namespace: String,
/// Value of the `openshell.sandbox_namespace` label applied to Docker sandboxes.
#[serde(alias = "sandbox_namespace")]
pub sandbox_label: String,

/// Gateway gRPC endpoint the sandbox connects back to.
pub grpc_endpoint: String,
Expand Down Expand Up @@ -147,7 +148,7 @@ impl Default for DockerComputeConfig {
socket_path: None,
default_image: openshell_core::image::default_sandbox_image(),
image_pull_policy: String::new(),
sandbox_namespace: "default".to_string(),
sandbox_label: "default".to_string(),
grpc_endpoint: String::new(),
supervisor_bin: None,
supervisor_image: None,
Expand All @@ -174,7 +175,7 @@ pub(crate) struct DockerGuestTlsPaths {
struct DockerDriverRuntimeConfig {
default_image: String,
image_pull_policy: String,
sandbox_namespace: String,
sandbox_label: String,
grpc_endpoint: String,
network_name: String,
gateway_route: DockerGatewayRoute,
Expand Down Expand Up @@ -456,7 +457,7 @@ impl DockerComputeDriver {
config: DockerDriverRuntimeConfig {
default_image: docker_config.default_image.clone(),
image_pull_policy: docker_config.image_pull_policy.clone(),
sandbox_namespace: docker_config.sandbox_namespace.clone(),
sandbox_label: docker_config.sandbox_label.clone(),
grpc_endpoint,
network_name,
gateway_route,
Expand Down Expand Up @@ -732,7 +733,7 @@ impl DockerComputeDriver {
);
self.publish_sandbox_snapshot(pending_sandbox_snapshot(
sandbox,
&self.config.sandbox_namespace,
&self.config.sandbox_label,
provisioning_condition(),
false,
));
Expand Down Expand Up @@ -1132,7 +1133,7 @@ impl DockerComputeDriver {
PendingSandboxRecord {
sandbox: pending_sandbox_snapshot(
sandbox,
&self.config.sandbox_namespace,
&self.config.sandbox_label,
provisioning_condition(),
false,
),
Expand Down Expand Up @@ -1187,7 +1188,7 @@ impl DockerComputeDriver {
cleanup_sandbox_token_file(sandbox, &self.config);
let snapshot = pending_sandbox_snapshot(
sandbox,
&self.config.sandbox_namespace,
&self.config.sandbox_label,
error_condition(failure.reason, &failure.message),
false,
);
Expand Down Expand Up @@ -1393,7 +1394,7 @@ impl DockerComputeDriver {
}

async fn list_managed_container_summaries(&self) -> Result<Vec<ContainerSummary>, Status> {
let filters = managed_container_label_filters(&self.config.sandbox_namespace, []);
let filters = managed_container_label_filters(&self.config.sandbox_label, []);
self.docker
.list_containers(Some(
ListContainersOptionsBuilder::default()
Expand All @@ -1418,7 +1419,7 @@ impl DockerComputeDriver {
}

let filters =
managed_container_label_filters(&self.config.sandbox_namespace, label_filter_values);
managed_container_label_filters(&self.config.sandbox_label, label_filter_values);
let containers = self
.docker
.list_containers(Some(
Expand All @@ -1436,7 +1437,7 @@ impl DockerComputeDriver {
};
let namespace_matches = labels
.get(LABEL_SANDBOX_NAMESPACE)
.is_some_and(|value| value == &self.config.sandbox_namespace);
.is_some_and(|value| value == &self.config.sandbox_label);
let id_matches = sandbox_id.is_empty()
|| labels
.get(LABEL_SANDBOX_ID)
Expand Down Expand Up @@ -2314,7 +2315,7 @@ fn sandbox_token_host_path_by_id(
) -> Result<PathBuf, Status> {
openshell_core::driver_utils::sandbox_token_path(
"docker-sandbox-tokens",
Some(&config.sandbox_namespace),
Some(&config.sandbox_label),
sandbox_id,
)
.map_err(|err| {
Expand Down Expand Up @@ -2664,13 +2665,13 @@ fn build_container_create_body_for_image(
LABEL_SANDBOX_WORKSPACE.to_string(),
sandbox.workspace.clone(),
);
// The list/get/find paths filter by `config.sandbox_namespace`, so use
// The list/get/find paths filter by `config.sandbox_label`, so use
// the same value here. `DriverSandbox.namespace` is unset on the request
// path (the gateway elides it), and using it would produce containers
// that the driver itself cannot find afterwards.
labels.insert(
LABEL_SANDBOX_NAMESPACE.to_string(),
config.sandbox_namespace.clone(),
config.sandbox_label.clone(),
);

Ok(ContainerCreateBody {
Expand Down Expand Up @@ -3256,12 +3257,12 @@ fn label_filters(values: impl IntoIterator<Item = String>) -> HashMap<String, Ve
}

fn managed_container_label_filters(
sandbox_namespace: &str,
sandbox_label: &str,
extra_values: impl IntoIterator<Item = String>,
) -> HashMap<String, Vec<String>> {
let mut values = vec![
format!("{LABEL_MANAGED_BY}={LABEL_MANAGED_BY_VALUE}"),
format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_namespace}"),
format!("{LABEL_SANDBOX_NAMESPACE}={sandbox_label}"),
];
values.extend(extra_values);
label_filters(values)
Expand Down
34 changes: 31 additions & 3 deletions crates/openshell-driver-docker/src/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
DockerDriverRuntimeConfig {
default_image: "image:latest".to_string(),
image_pull_policy: String::new(),
sandbox_namespace: "default".to_string(),
sandbox_label: "default".to_string(),
grpc_endpoint: "https://localhost:8443".to_string(),
network_name: DEFAULT_DOCKER_NETWORK_NAME.to_string(),
gateway_route: DockerGatewayRoute::Bridge {
Expand Down Expand Up @@ -123,6 +123,34 @@ fn runtime_config() -> DockerDriverRuntimeConfig {
}
}

#[test]
fn docker_config_uses_canonical_sandbox_label_name() {
let config: DockerComputeConfig =
serde_json::from_value(serde_json::json!({ "sandbox_label": "tenant-a" })).unwrap();
assert_eq!(config.sandbox_label, "tenant-a");

let serialized = serde_json::to_value(config).unwrap();
assert_eq!(serialized["sandbox_label"], "tenant-a");
assert!(serialized.get("sandbox_namespace").is_none());
}

#[test]
fn docker_config_accepts_legacy_sandbox_namespace_alias() {
let config: DockerComputeConfig =
serde_json::from_value(serde_json::json!({ "sandbox_namespace": "tenant-a" })).unwrap();
assert_eq!(config.sandbox_label, "tenant-a");
}

#[test]
fn docker_config_rejects_canonical_and_legacy_sandbox_label_names_together() {
let error = serde_json::from_value::<DockerComputeConfig>(serde_json::json!({
"sandbox_label": "tenant-a",
"sandbox_namespace": "tenant-b"
}))
.expect_err("canonical and legacy names must not both be accepted");
assert!(error.to_string().contains("duplicate field"));
}

fn json_struct(value: serde_json::Value) -> prost_types::Struct {
let serde_json::Value::Object(object) = value else {
panic!("expected JSON object");
Expand Down Expand Up @@ -2041,10 +2069,10 @@ fn build_container_create_body_uses_runtime_namespace_label() {
// runtime config, not from `DriverSandbox.namespace`. The gateway
// does not populate `DriverSandbox.namespace`, so a container created
// with that empty value would not match subsequent list/get/find
// queries (which filter on `config.sandbox_namespace`), leaking
// queries (which filter on `config.sandbox_label`), leaking
// sandboxes that the driver itself cannot observe.
let mut config = runtime_config();
config.sandbox_namespace = "tenant-a".to_string();
config.sandbox_label = "tenant-a".to_string();
let mut sandbox = test_sandbox();
sandbox.namespace = "ignored-by-driver".to_string();

Expand Down
14 changes: 6 additions & 8 deletions crates/openshell-driver-podman/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,9 +247,9 @@ Podman follows the same end-to-end contract as the Kubernetes and VM drivers
for the in-container SSH relay: gateway config to `PodmanComputeConfig` to
sandbox environment to supervisor session registration on that path.

1. `openshell-core` `Config::sandbox_ssh_socket_path` is copied into
`PodmanComputeConfig::sandbox_ssh_socket_path` when the gateway builds the
in-process driver.
1. `[openshell.drivers.podman].ssh_socket_path` is deserialized into
`PodmanComputeConfig::ssh_socket_path` when the gateway builds the in-process
driver. The field defaults to `/run/openshell/ssh.sock` when omitted.
2. `build_env()` in `container.rs` sets `OPENSHELL_SSH_SOCKET_PATH` to that
value, alongside required vars such as `OPENSHELL_ENDPOINT` and
`OPENSHELL_SANDBOX_ID`. These driver-controlled entries overwrite template
Expand Down Expand Up @@ -430,11 +430,9 @@ matter compared to cluster or rootful runtimes:

- Gateway integration: `crates/openshell-server/src/compute/mod.rs`
(`new_podman` and `PodmanComputeDriver` wiring).
- Server configuration: `crates/openshell-server/src/lib.rs`
(`ComputeDriverKind::Podman` builds `PodmanComputeConfig` including
`sandbox_ssh_socket_path` from gateway `Config`).
- Gateway relay path: `openshell-core` `Config::sandbox_ssh_socket_path` in
`crates/openshell-core/src/config.rs`.
- Server configuration:
`crates/openshell-server/src/compute/driver_config/builtin.rs` builds
`PodmanComputeConfig` from `[openshell.drivers.podman]`.
- SSRF mitigation: `crates/openshell-core/src/net.rs`,
`crates/openshell-sandbox/src/proxy.rs`, and
`crates/openshell-server/src/grpc/policy.rs`.
Expand Down
Loading
Loading