EAI-5958: Add kubeletstats collector to stamp workload_id on pod CPU/mem metrics - #810
Merged
Merged
Conversation
…mem metrics Per-pod CPU and memory time-series are needed for EPYC (CPU-only) AIM observability in the AIWB dashboard, scoped the same way GPU/VRAM metrics already are: by a stable workload_id label. cAdvisor metrics (scraped by otel-collector-metrics-k8s via a node-role scrape) are node-scoped, so pod/namespace arrive as datapoint labels and cannot be associated to a pod by the k8sattributes processor, which works at the resource level. As a result container_cpu_usage_seconds_total / container_memory_working_set_bytes carry no workload_id. Add a dedicated daemonset collector using the kubeletstats receiver, which emits pod/container-scoped metrics with k8s.pod.* as resource attributes. k8sattributes then looks up the pod and stamps its airm.silogen.ai/workload-id (and project-id) label; a transform copies the identity resource attributes down to datapoint attributes so they survive OTLP ingestion as series labels (workload_id, project_id, pod, namespace, container). Reuses the existing otel-collector ServiceAccount (RBAC already grants nodes/stats).
The kubeletstats receiver used endpoint https://${env:K8S_NODE_NAME}:10250 with
K8S_NODE_NAME from spec.nodeName. Cluster DNS does not resolve node hostnames, so
every scrape failed with 'no such host' and the collector produced no metrics.
Use status.hostIP (always routable) instead.
…rations - Drop the 'container' kubeletstats metric group: consumers query the k8s_pod_* series, so the container group only added per-container cardinality (overlapping cAdvisor). - Add catch-all tolerations so the daemonset runs on tainted nodes (GPU/inference), collecting per-workload CPU/memory fleet-wide like the node-exporter daemonset.
Add filter.node_from_env_var so each daemonset pod's k8sattributes processor watches only its own node's pods (cost scales with pods-per-node, not pods x nodes). kubeletstats already scrapes only the local node, so local pod metadata is all the enrichment needs. Re-adds a K8S_NODE_NAME env used solely as the local watch-filter key (not for DNS).
…ersion) The kubeletstats collector added in the preceding commits landed in otel-lgtm-stack/v1.0.7, but root/values.yaml deploys v1.0.8 — so the per-pod CPU/memory workload_id enrichment was inert. Add the same daemonset collector to v1.0.8, wiring memory_limiter and the kubelet collection_interval to the .Values keys v1.0.8 parameterized (rather than the hardcoded settings v1.0.7 used) so it matches the other collectors in this chart.
nowycondro
approved these changes
Aug 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds the EAI-5958 kubeletstats collector that stamps
workload_id(andproject_id) onto per-pod CPU/memory metrics, so per-workload CPU/mem observability matches how GPU and vLLM metrics already carryworkload_id. Cherry-picked from #808 (still open) and ported to the deployed chart version.Why a dedicated kubeletstats daemonset
The cAdvisor metrics scraped by
otel-collector-metrics-k8sare node-scoped, so pod/namespace arrive as datapoint labels and can't be associated to a pod by thek8sattributesprocessor (which works at the resource level). This daemonset uses the kubeletstats receiver, which emits pod/container-scoped metrics withk8s.pod.*as resource attributes — sok8sattributescan look up the pod and stamp itsairm.silogen.ai/workload-idlabel. Atransformthen copies those identity resource attributes down to datapoint attributes so they survive OTLP ingestion as series labels (workload_id,project_id,pod,namespace,container).Contents
v1.0.7— the four original EAI-5958 commits from Envoy collector scale to zero #808, cherry-picked unchanged (kubeletstats collector; kubelet addressed by node IPstatus.hostIPnot node name; review fixes — pod-only metric group +tolerations: Exists; node-scopedk8sattributeswatch).v1.0.8— the same collector ported to the active version.root/values.yamldeploysotel-lgtm-stack/v1.0.8, so thev1.0.7change alone would be inert (as it is on Envoy collector scale to zero #808 today). The port wiresmemory_limiterand the kubeletcollection_intervalto the.Valueskeys v1.0.8 parameterized (collectors.memoryLimiter.*,collectors.scrapeInterval), matching every other collector in that chart.Non-obvious decisions
${env:K8S_NODE_NAME}:10250fails with "no such host";status.hostIPis always routable.K8S_NODE_NAMEis kept only as the localk8sattributeswatch filter (not for DNS).k8sattributeswatch — each daemonset pod watches only its own node's pods (node_from_env_var: K8S_NODE_NAME), so cache cost scales with pods-per-node instead of pods × nodes.tolerations: Exists— runs on every node including tainted GPU/inference nodes, so per-workload CPU/mem is collected fleet-wide (matches the node-exporter daemonset's scheduling).Test plan
helm template sources/otel-lgtm-stack/v1.0.8renders clean (exit 0); parameterized values substitute correctly in the new collector.helm lint sources/otel-lgtm-stack/v1.0.8passes.otel-collector-metrics-kubeletdaemonset runs on every node and thatk8s_pod_*CPU/memory series carry theworkload_id/project_idlabels in Grafana/LGTM.