[FEATURE] helm/kagent: expose OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME, and OTEL_RESOURCE_ATTRIBUTES

[nirzOps]

📝 Feature Summary

Add first-class values.yaml knobs in helm/kagent for the standard OpenTelemetry env vars the controller already reads via the OTEL SDK but that the chart doesn't currently render: OTEL_EXPORTER_OTLP_TRACES_HEADERS, OTEL_EXPORTER_OTLP_LOGS_HEADERS, OTEL_SERVICE_NAME, and OTEL_RESOURCE_ATTRIBUTES.

❓ Problem Statement / Motivation

The kagent controller in go/core/internal/telemetry/tracing.go uses autoexport.NewSpanExporter(ctx) and resource.WithFromEnv(), both of which honor the full set of standard OTEL SDK env vars:

• OTEL_EXPORTER_OTLP_HEADERS / OTEL_EXPORTER_OTLP_TRACES_HEADERS / OTEL_EXPORTER_OTLP_LOGS_HEADERS
• OTEL_SERVICE_NAME
• OTEL_RESOURCE_ATTRIBUTES

But the Helm chart's otel.tracing.exporter.otlp schema in helm/kagent/values.yaml only surfaces endpoint, protocol, timeout, insecure. This is a gap because every managed OTLP backend that requires authenticated ingest is unreachable without headers:

Backend	Auth mechanism	Header required
Langfuse Cloud	Basic auth over OTLP	Authorization: Basic <b64(publicKey:secretKey)>
Honeycomb	API key	x-honeycomb-team: <key>
Datadog	API key	dd-api-key: <key>
Grafana Cloud	Basic auth	Authorization: Basic <b64(id:token)>
New Relic	License key	api-key: <license>

Similarly, OTEL_SERVICE_NAME / OTEL_RESOURCE_ATTRIBUTES are how the OTEL spec expects service.name, deployment.environment.name, tenant/team tags, etc. to flow into resource attributes — for tools like Langfuse that use langfuse.environment to partition traces by env, these need to be settable per install.

Today the only way to plumb any of these through is to bypass otel.* entirely and inject env vars via controller.envFrom from a hand-crafted ConfigMap + Secret maintained outside the chart. That works but it's a fork/wrapper-chart-only pattern — you can't opt into managed tracing declaratively from helm install.

Concrete downstream example of that workaround: elementor/elementor-cloud-external-charts@1e8b9436 plus elementor/elementor-agents@13545eb, which together maintain a parallel kagent-langfuse-otel ConfigMap and kagent-langfuse-auth ExternalSecret and wire them through kagent.controller.envFrom.

💡 Proposed Solution

Extend helm/kagent/values.yaml under the existing otel: block:

otel:
  # NEW
  serviceName: ""              # -> OTEL_SERVICE_NAME
  resourceAttributes: {}       # map -> OTEL_RESOURCE_ATTRIBUTES (k=v,k=v)
  tracing:
    enabled: false
    exporter:
      otlp:
        endpoint: ""
        protocol: "grpc"
        timeout: 15000
        insecure: true
        headers: {}            # NEW: map -> OTEL_EXPORTER_OTLP_TRACES_HEADERS
  logging:
    enabled: false
    exporter:
      otlp:
        endpoint: ""
        timeout: 15000
        insecure: true
        headers: {}            # NEW: map -> OTEL_EXPORTER_OTLP_LOGS_HEADERS

Rendering rules in helm/kagent/templates/controller-configmap.yaml:

• otel.serviceName — rendered only when non-empty (backward compatible).
• otel.resourceAttributes / otel.tracing.exporter.otlp.headers / otel.logging.exporter.otlp.headers — rendered only when non-empty, serialized as key1=value1,key2=value2 with sorted keys (deterministic output, same pattern already used for DEFAULT_AGENT_POD_LABELS).

Secret headers (Langfuse Basic auth, Honeycomb API key, ...) should still be overlaid at the pod level via the existing controller.envFrom — no new secret-injection surface is added. The chart-provided data.OTEL_EXPORTER_OTLP_TRACES_HEADERS is transparently overridden when controller.envFrom supplies the same env var, so both patterns compose cleanly. The values.yaml comments document this.

Design decisions:

1. No new secretRef schema. OTLP header secrets are a two-line envFrom in existing chart surface; adding another schema (headersSecretRef) would duplicate the extension point.
2. Symmetric traces/logs. Both tracing.exporter.otlp.headers and logging.exporter.otlp.headers are added — the chart already treats both endpoints symmetrically for insecure/timeout, so headers follows.
3. OTEL_SERVICE_NAME at the otel.* root, not under tracing.*. Per OTEL spec it's a global resource attribute, not per-signal.
4. Backward compatible. With defaults unchanged, helm template output is byte-identical.

🔄 Alternatives Considered

• Only add headers, punt on serviceName / resourceAttributes. Rejected — resourceAttributes is the standard way to tag traces with env/tenant for backends like Langfuse (which partitions by langfuse.environment).
• Add otel.tracing.exporter.otlp.headersSecretRef: {name, key} for native secret refs. Rejected for the first PR — controller.envFrom already covers this and adding a parallel schema increases surface area for little gain. Happy to add if maintainers prefer.
• Native Langfuse subchart / CRD. Out of scope — this is a generic OTLP capability, Langfuse is just one consumer.

🎯 Affected Service(s)

Controller Service (chart-only change to helm/kagent; no controller code change — controller already reads these env vars via the standard OTEL SDK).

📚 Additional Context

• Prior OTLP work: [BUG] Python OTLP exporter hardcodes gRPC, ignores OTEL_EXPORTER_OTLP_PROTOCOL #1681 (protocol honored), fix: propagate trace context end-to-end for agent Services #1297 (trace context propagation).
• Draft PR is coming from nirzOps/kagent:feat/otel-headers-service-name-resource-attrs. Will cross-link once opened.
• Design discussion in Ideas category: (link coming right after this issue).
• Happy to sign the DCO / follow the "large change" flow from CONTRIBUTING.md — I'll ping #kagent-dev on CNCF Slack after filing.

🤝 Contribution

I'd like to contribute the PR myself.

[nirzOps]

Follow-ups now live:

• Design discussion: helm/kagent: declarative OTLP headers + service name + resource attributes (unblocks Langfuse Cloud / Honeycomb / Datadog / ...) #2127
• Draft PR: feat(helm): expose OTEL headers, service name, and resource attributes #2128

Slack ping to #kagent-dev on CNCF Slack is pending on my side.