feat: add authenticated OpenTelemetry gateway - #231
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds an authenticated OpenTelemetry Collector with SAK-to-JWT enrollment, optional Helm deployment and agent routing, integration validation, multi-architecture image publishing, and nested-module build checks. ChangesOpenTelemetry gateway
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Agent as fleet-intelligence-agent
participant Gateway as fleetint-otelcol
participant Enrollment as enrollment endpoint
participant Backend as OTLP backend
Agent->>Gateway: Send OTLP metrics and logs
Gateway->>Enrollment: Enroll with SAK bearer token
Enrollment-->>Gateway: Return JWT assertion
Gateway->>Backend: Export OTLP with JWT and Nv-Actor-Id
Backend-->>Gateway: Return response or refreshed JWT
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 188-189: Update the actions/checkout@v4 step in the build job to
set persist-credentials to false, ensuring the checkout action does not store
the GITHUB_TOKEN in git configuration before third-party tooling runs.
- Around line 191-197: Update the “Set up Go” workflow step to use
actions/setup-go@v6 instead of v4, keeping its existing go-version configuration
unchanged.
In `@cmd/fleetint/run.go`:
- Around line 225-227: Update the FLEETINT_COLLECTOR_ENDPOINT handling in the
environment override block so Infow does not log the raw endpoint value. Log
only that the collector endpoint override was enabled, or pass a safely redacted
endpoint while preserving assignment of the full value to he.CollectorEndpoint.
In `@internal/config/config.go`:
- Around line 181-190: Update the collector_endpoint validation around
parsed.ForceQuery and the raw endpoint string to reject bare trailing ? and #
characters, while preserving the existing credential, query, fragment, and
HTTP(S) checks. Add regression tests covering http://collector.example? and
http://collector.example#.
In `@otelcol/auth/sakauth/config.go`:
- Around line 57-59: Update the SAK enrollment TLS validation in the
configuration validation method containing the c.TLS.Insecure check to also
reject c.TLS.InsecureSkipVerify. Return the same validation error for either
insecure TLS setting, ensuring enrollment requires HTTPS certificate
verification.
In `@otelcol/auth/sakauth/extension.go`:
- Around line 246-252: Guard the response-token update in the request flow
around the jwt_assertion handling so it only applies when the shared rt.ext.jwt
still equals the request’s usedJWT; otherwise leave the newer token and
customerID unchanged. Add a regression test covering concurrent or out-of-order
responses, including a 401 refresh, to verify stale responses cannot overwrite
the newest JWT.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9c891f07-c5ab-4125-b79f-f13e6e4bb1f6
⛔ Files ignored due to path filters (1)
otelcol/auth/sakauth/go.sumis excluded by!**/*.sum
📒 Files selected for processing (29)
.dockerignore.github/workflows/ci.yml.github/workflows/release.yml.gitignoreMakefilecmd/fleetint/gateway_test.gocmd/fleetint/metadata.gocmd/fleetint/run.godeployments/helm/fleet-intelligence-agent/templates/daemonset.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-configmap.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-deployment.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-networkpolicy.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-pdb.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-secret.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-service.yamldeployments/helm/fleet-intelligence-agent/values.yamldeployments/otel-collector/config.yamlinternal/config/config.gointernal/config/config_test.gointernal/exporter/exporter.gointernal/exporter/exporter_test.gootelcol/Dockerfileotelcol/auth/sakauth/collector_integration_test.gootelcol/auth/sakauth/config.gootelcol/auth/sakauth/extension.gootelcol/auth/sakauth/extension_test.gootelcol/auth/sakauth/factory.gootelcol/auth/sakauth/go.modotelcol/otelcol-builder.yaml
jingxiang-z
reviewed
Jul 20, 2026
Add an optional custom OpenTelemetry Collector that centralizes SAK-to-JWT authentication and forwards agent metrics and logs while preserving direct enrollment for inventory and attestation. Harden the gateway for production with current Collector dependencies, secure token and TLS handling, highly available Helm resources, GHCR publishing, and end-to-end OTLP integration coverage. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
otelcol/auth/sakauth has its own go.mod, so root-module `./...` targets never descended into it: its unit and config tests, golangci-lint and govulncheck were all silently skipped. Add an OTELCOL_MODULE variable and extend the test, lint and vuln targets to run there too. Keeping this in the Makefile means `make docker-test` covers the module as well, since it runs those three targets. The CI lint job invokes golangci-lint directly rather than through make, so it gets a matching step. Also set persist-credentials: false on the build_check_otelcol checkout, which installs and runs the OTel Collector Builder and its dependencies. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
Validate() rejected tls.insecure but not tls.insecure_skip_verify, which kept the https:// scheme and so passed every other check while disabling certificate verification. The jwt_assertion handler also overwrote the stored token unconditionally, so an out-of-order response could replace a newer token with an older one. Install the refreshed token only while the stored one is unchanged. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
The FLEETINT_COLLECTOR_ENDPOINT override logged the raw value, which can carry credentials in userinfo or a token in the path. Validation rejects those but runs later, so log only the host. Validation also missed a bare trailing "?" and "#": url.Parse reports the first as ForceQuery with an empty RawQuery and drops the second entirely, so both reached the point where "/v1/metrics" is appended. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
The collector memory_limiter is configured by percentage and derives that percentage from the container's cgroup memory limit, falling back to total node memory when none is set. On a large node an unlimited pod would size the limiter against the node and be OOM-killed before it ever engages, so reject that at render time. Document the relationship in values.yaml: the percentages track resources.limits.memory automatically, so raising the pod limit needs no second edit. Delete deployments/otel-collector/config.yaml. It was unreferenced by the Dockerfile, the builder manifest and the tests, and its hardcoded limit_mib contradicted the ConfigMap the chart actually deploys. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
rsampaio
force-pushed
the
rsampaio-otel-gateway
branch
from
July 28, 2026 21:40
0ac718d to
4e2bb61
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
.github/workflows/release.yml (1)
142-150: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider embedding version metadata into the otelcol image.
Unlike the agent image build (Lines 128-141), this step passes no
build-args/OCI labels, so the publishedfleetint-otelcolimage carries no embedded version/revision/build-timestamp for traceability. Would need correspondingARG/LABELadditions inotelcol/Dockerfile.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/release.yml around lines 142 - 150, Update the “Build and push fleetint-otelcol image” step to pass the same version, revision, and build-timestamp metadata build arguments and OCI labels used by the agent image, and add the corresponding ARG/LABEL declarations in otelcol/Dockerfile so the published image embeds that metadata.otelcol/Dockerfile (1)
9-9: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider pinning base images by digest.
golang:1.26.5andgcr.io/distroless/static:nonrootare tag-referenced rather than digest-pinned, so the same tag can resolve to different content over time, weakening build reproducibility for a multi-arch, publicly-published image.Also applies to: 27-27
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@otelcol/Dockerfile` at line 9, Update the Dockerfile’s builder and runtime base image references, including golang:1.26.5 and gcr.io/distroless/static:nonroot, to use immutable image digests while retaining their current tags and multi-architecture support.deployments/helm/fleet-intelligence-agent/templates/otel-gateway-deployment.yaml (1)
85-88: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winHardcoded secret key name for
existingSecret.When
otelGateway.enroll.existingSecretis set, the key is hardcoded tosak-token. Operators bringing an existing Secret (the recommended production path per the PR's objectives) may already use a different key name, forcing them to either rename/duplicate the secret or fail silently.♻️ Suggested addition of a configurable key field
- name: SAK_TOKEN valueFrom: secretKeyRef: name: {{ .Values.otelGateway.enroll.existingSecret | default (printf "%s-otel-gateway-sak" (include "fleet-intelligence-agent.fullname" .)) }} - key: sak-token + key: {{ .Values.otelGateway.enroll.existingSecretKey | default "sak-token" }}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@deployments/helm/fleet-intelligence-agent/templates/otel-gateway-deployment.yaml` around lines 85 - 88, Make the secret key reference in the otel gateway deployment configurable for existing secrets instead of always using sak-token. Add or reuse an otelGateway enrollment values field for the key, use it in the secretKeyRef under the existingSecret path, and preserve sak-token as the default when no override is provided.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@otelcol/auth/sakauth/extension.go`:
- Around line 148-155: Update refreshJWT’s post-performEnrollment write to
compare e.jwt with the previously captured staleJWT while holding e.mu; install
the enrollment JWT and customer ID only when they still match, otherwise return
the current e.jwt without overwriting it. Add a regression test covering an
in-flight enrollment refresh completing after a newer response-header refresh,
asserting the newer token remains active.
---
Nitpick comments:
In @.github/workflows/release.yml:
- Around line 142-150: Update the “Build and push fleetint-otelcol image” step
to pass the same version, revision, and build-timestamp metadata build arguments
and OCI labels used by the agent image, and add the corresponding ARG/LABEL
declarations in otelcol/Dockerfile so the published image embeds that metadata.
In
`@deployments/helm/fleet-intelligence-agent/templates/otel-gateway-deployment.yaml`:
- Around line 85-88: Make the secret key reference in the otel gateway
deployment configurable for existing secrets instead of always using sak-token.
Add or reuse an otelGateway enrollment values field for the key, use it in the
secretKeyRef under the existingSecret path, and preserve sak-token as the
default when no override is provided.
In `@otelcol/Dockerfile`:
- Line 9: Update the Dockerfile’s builder and runtime base image references,
including golang:1.26.5 and gcr.io/distroless/static:nonroot, to use immutable
image digests while retaining their current tags and multi-architecture support.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: d4079e01-ec41-48da-bf03-7f384b296552
⛔ Files ignored due to path filters (1)
otelcol/auth/sakauth/go.sumis excluded by!**/*.sum
📒 Files selected for processing (28)
.dockerignore.github/workflows/ci.yml.github/workflows/release.yml.gitignoreMakefilecmd/fleetint/gateway_test.gocmd/fleetint/metadata.gocmd/fleetint/run.godeployments/helm/fleet-intelligence-agent/templates/daemonset.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-configmap.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-deployment.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-networkpolicy.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-pdb.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-secret.yamldeployments/helm/fleet-intelligence-agent/templates/otel-gateway-service.yamldeployments/helm/fleet-intelligence-agent/values.yamlinternal/config/config.gointernal/config/config_test.gointernal/exporter/exporter.gointernal/exporter/exporter_test.gootelcol/Dockerfileotelcol/auth/sakauth/collector_integration_test.gootelcol/auth/sakauth/config.gootelcol/auth/sakauth/extension.gootelcol/auth/sakauth/extension_test.gootelcol/auth/sakauth/factory.gootelcol/auth/sakauth/go.modotelcol/otelcol-builder.yaml
🚧 Files skipped from review as they are similar to previous changes (15)
- .gitignore
- cmd/fleetint/metadata.go
- deployments/helm/fleet-intelligence-agent/templates/daemonset.yaml
- internal/config/config_test.go
- otelcol/otelcol-builder.yaml
- otelcol/auth/sakauth/config.go
- cmd/fleetint/run.go
- internal/exporter/exporter_test.go
- internal/config/config.go
- internal/exporter/exporter.go
- otelcol/auth/sakauth/factory.go
- deployments/helm/fleet-intelligence-agent/values.yaml
- .dockerignore
- Makefile
- otelcol/auth/sakauth/extension_test.go
Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
rsampaio
force-pushed
the
rsampaio-otel-gateway
branch
from
July 28, 2026 22:26
e9306b6 to
d679bcc
Compare
jingxiang-z
approved these changes
Aug 4, 2026
Envoy injects this header from the verified JWT claim at the backend ingress, so setting it here was an unverified duplicate that got overwritten anyway. Removes extractCustomerID and the customerID field; tests now assert the header is absent. Signed-off-by: Rodrigo Sampaio Vaz <rvaz@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds an optional, production-hardened OpenTelemetry gateway deployment for Fleet Intelligence. Agents route OTLP metrics and logs through a per-cluster custom Collector, which owns backend SAK-to-JWT authentication; direct agent enrollment remains unchanged for inventory and attestation.
The gateway is disabled by default and does not change bare-metal or existing Kubernetes behavior unless
otelGateway.enabled=true.Security and reliability
Review guide
cmd/fleetint/run.go,internal/config/, andinternal/exporter/establish collector precedence without forwarding node credentials.otelcol/auth/sakauth/contains enrollment, token refresh, TLS, and request authentication.deployments/helm/fleet-intelligence-agent/templates/otel-gateway-*contains availability and security controls.otelcol/Dockerfile,otelcol/otelcol-builder.yaml, and GitHub workflows build, test, and publish the collector.collector_integration_test.gostarts the built Collector and verifies real OTLP metrics/logs forwarding, identity attributes, and backend authentication.Operational notes
otelGateway.enroll.existingSecretrather than passing a SAK through Helm values.Test plan
90.8%statement coverage).go vetand fullgolangci-lint(0 issues).Checklist
Summary by CodeRabbit
fleetint-otelcolimage, including automated publishing/tagging.