Do not report security vulnerabilities through a public GitHub issue, pull request, discussion, or other public channel.
Use one of these private reporting channels:
- Web: NVIDIA Vulnerability Disclosure Program (preferred)
- Email: psirt@nvidia.com. Use the NVIDIA public PGP key for encrypted communication.
- GitHub: Use the repository Security tab and select Report a vulnerability.
Include:
- The affected NVCF version, release, branch, or commit
- The affected component and vulnerability type
- Reproduction steps and required configuration
- Proof-of-concept code, if available
- The expected and observed behavior
- An impact assessment, including affected security boundaries
NVIDIA Product Security Incident Response Team (PSIRT) will acknowledge the report, validate the issue and severity, coordinate a fix, and publish a security bulletin when appropriate.
NVIDIA Cloud Functions (NVCF) is a Kubernetes-based platform for deploying, managing, and invoking GPU workloads. This repository contains service code, CLIs, shared libraries, deployment charts, migrations, examples, and validation tooling.
NVCF has three primary planes:
- The control plane manages function, task, deployment, secret, and cluster state.
- The invocation plane authenticates and routes HTTP, streaming, and gRPC requests to function workloads and applies rate limits.
- The compute plane connects GPU clusters to the platform. NVIDIA Cluster Agent (NVCA) turns platform requests into Kubernetes resources and manages workload lifecycle.
NATS JetStream connects these planes and buffers work. Kubernetes, artifact registries, object storage, secret stores, telemetry backends, and platform data stores are external security dependencies.
Repository Exposure Classification: Public.
Basis: NVIDIA/nvcf is a public repository on GitHub. This document uses
public-safe detail.
Service Exposure Classification: External / Regulated (high confidence). Basis: NVCF exposes customer-facing APIs, runs production GPU workloads, and handles credentials, workload artifacts, request and response data, secrets, and telemetry.
- External callers cross an ingress boundary into the control-plane and invocation APIs.
- Invocation services cross an authorization boundary when they validate a caller and resolve the function versions that caller may invoke.
- Control-plane and invocation services cross a messaging boundary when they publish work to NATS JetStream.
- NVCA crosses a cluster-administration boundary when it creates and manages pods, Helm releases, custom resources, service accounts, and related Kubernetes objects.
- User-supplied function and task containers run inside the compute plane but are not trusted as platform components.
- Artifact, secret, registry, object-storage, and telemetry integrations cross deployment-specific network and credential boundaries.
This threat model incorporates the repository architecture, current service implementations, and an existing STRIDE assessment. Threats are ordered by expected impact and exposure.
-
Authorization and tenant-routing confusion
The HTTP invocation service protects legacy invocation routes retained only for backward compatibility, transparent load-balancer routes, and worker attach routes in
src/invocation-plane-services/http-invocation/crates/server/src/app.rs. The legacy routes are deprecated and must not be used for new integrations. It delegates function authorization throughnvcf_api/mod.rs. The gRPC proxy independently uses bearer tokens,function-id, function-version metadata, request IDs, and connection state insrc/invocation-plane-services/grpc-proxy/proxy/director.go. Incorrectly binding any token, function identifier, cached authorization result, request ID, or worker connection could let one caller invoke or reconnect to another tenant's workload. -
Privileged workload orchestration
NVCA consumes platform work messages and manages
ICMSRequest,MiniService, pod, and Helm workload state in GPU clusters. A forged, replayed, or incorrectly authorized launch specification could cause unintended images, charts, service accounts, volumes, or Kubernetes resources to run with compute-plane privileges. Queue authentication, namespace isolation, admission policy, and least-privilege RBAC are critical controls. -
Secret and credential disclosure
Invocation bearer tokens, NATS and OAuth credentials, rate-limiter tokens, registry credentials, object-storage URLs, and workload secrets pass through multiple services. Relevant paths include
src/invocation-plane-services/http-invocation/crates/server/src/middleware/auth.rs,src/invocation-plane-services/grpc-proxy/proxy/credentials/bearer.go,src/control-plane-services/nats-auth-callout, the OpenBao deployment assets, andsrc/compute-plane-services/image-credential-helper. Logging, tracing, error handling, configuration output, or telemetry export that records these values could expose access to platform or tenant resources. -
Request, response, and artifact data exposure or tampering
The invocation service can buffer or stream request bodies, forward headers through NATS, issue worker attach tokens, and use object storage for assets and large responses.
src/compute-plane-services/worker-init/internal/downloader/downloader.godownloads workload artifacts from credential-bearing URLs. A stolen or misbound attach token or signed URL could expose customer data. A tampered request, response, model, container, or Helm artifact could alter workload behavior or execute attacker-controlled code. -
Resource exhaustion and rate-limit bypass
Public HTTP, streaming, gRPC CONNECT, status, and worker attach paths can hold connections, buffer data, enqueue work, trigger autoscaling, and consume GPU capacity. Disabled, unavailable, incorrectly scoped, or bypassed rate limiting could permit denial of service or unexpected cost. Request-size, connection, queue, timeout, and per-tenant workload limits must remain effective together.
-
Telemetry data exposure
src/compute-plane-services/byoo-otel-collectorprocesses workload logs, metrics, and traces, including log bodies and Kubernetes metadata. Misconfigured exporters, overly broad metadata collection, unsafe log content, or weak backend access control could disclose tenant data, credentials, model details, or operational topology. -
Deployment and software supply-chain tampering
Bazel rules, container image definitions, Helm and Helmfile configuration, migrations, generated manifests, and release automation determine what runs in control-plane and compute-plane clusters. A compromised dependency, registry credential, image tag, chart source, migration, or release workflow could introduce code or configuration across many NVCF deployments.
- The configured identity provider issues valid, short-lived credentials, and authorization services correctly bind callers to organizations, functions, function versions, and administrative actions.
- Production ingress terminates TLS with current protocols and certificates, filters abusive traffic, and exposes only intended service routes.
- NATS JetStream, platform data stores, object storage, registries, secret stores, and telemetry backends authenticate peers, encrypt traffic, enforce least privilege, and protect data at rest.
- Kubernetes control planes, nodes, container runtimes, RBAC, admission controls, network policies, and namespace boundaries isolate platform components and mutually untrusted workloads.
- User-provided container images and Helm charts are untrusted relative to the platform. Deployments apply the policies needed to prevent workload access to host resources, other tenants, and platform credentials.
- Images, charts, binaries, models, and other artifacts come from approved sources and retain their expected integrity between publication and use.
- Worker attach tokens, signed object-storage URLs, session identifiers, and cached authorization decisions remain short-lived and bound to the intended request, caller, function, and deployment.
- Operators enable and correctly scope rate limits, timeouts, quotas, and queue limits for production traffic.
- Health, metrics, profiling, debugging, and administrative endpoints are restricted to authorized operator networks and are disabled when unnecessary.
- Applications avoid placing secrets or sensitive request data in logs, traces, metrics, error messages, or other telemetry.
- NVCF maintainers, signed release automation, and authorized deployment operators are trusted to publish and configure platform components.
- Authenticated callers are trusted only for the organizations, functions, and actions granted by the authorization service.
- Function and task images, charts, request bodies, headers, model artifacts, and workload-generated telemetry are untrusted inputs.
- NVCA and other cluster controllers are privileged components. Their NATS identities, Kubernetes service accounts, and reconciliation inputs must be tightly scoped.
- Cloud services, Kubernetes, NATS, registries, secret stores, object storage, and telemetry systems are conditionally trusted dependencies. NVCF relies on their configured identity, encryption, isolation, durability, and audit controls.
Self-managed operators are responsible for:
- Configuring TLS, ingress filtering, network policy, and private access for internal service endpoints
- Supplying least-privilege identities for NATS, Kubernetes, registries, object storage, data stores, and secret stores
- Enabling authentication, authorization, rate limiting, quotas, and audit forwarding for production environments
- Pinning and verifying approved image, chart, and dependency versions
- Rotating credentials and limiting access to secret-bearing files and Kubernetes Secrets
- Restricting workload egress and preventing access to node, metadata-service, and platform control credentials
- Protecting telemetry pipelines and applying retention, redaction, and access policies appropriate for customer data
These notes help distinguish deployable security impact from repository-only test and documentation content. They do not suppress findings by themselves.
- Files under
examples/, test directories, and documentation may contain placeholder credentials, sample payloads, local endpoints, or intentionally insecure configurations used to demonstrate a boundary. Confirm a production path before treating them as deployed secrets or controls. - Files under
vendor/are third-party source snapshots. Assess findings against the version used by a shipped component and the reachable call path. - Local-development configuration may disable TLS, authentication, or rate limiting to support isolated testing. Such settings are unsafe if promoted to a production deployment.
- Security issues in user-provided workload code are normally owned by that workload. They are NVCF issues when the platform breaks an isolation, authorization, confidentiality, integrity, or availability boundary.
- Use short-lived credentials and least-privilege service accounts.
- Keep authorization, rate limiting, and worker-connection checks fail-safe for protected operations.
- Pin production artifacts by immutable version or digest and scan dependencies, images, and charts before release.
- Restrict NATS subjects, Kubernetes namespaces, secret paths, registry scopes, and object-storage objects to the smallest required audience.
- Redact authorization headers, signed URLs, request bodies, model data, and secrets from logs and telemetry.
- Monitor authentication failures, denied rate-limit checks, queue growth, unusual autoscaling, secret access, and privileged Kubernetes changes.
- Re-run the repository security review after material architecture, deployment, authentication, data-flow, or trust-boundary changes.