Add performance test suite for BYOO Collector

[sbaum1994]

Description

Create a repeatable performance test suite for the BYOO OpenTelemetry collector. The suite must deploy the collector with the same Kubernetes workload shapes produced for NVCF functions through the shared src/libraries/go/lib/pkg/icms-translate library.

The suite must support local development on k3d and execution against an existing remote Kubernetes cluster. It must establish reproducible performance baselines for logs and metrics without adding pass/fail regression thresholds in the initial version.

Deployment environments

• Support a managed local mode that creates a dedicated k3d cluster.
• Support a remote mode that deploys to an existing Kubernetes context without provisioning or modifying the cluster.
• Use an isolated namespace and labels so test resources can be identified and removed.
• Clean up test resources by default and provide an option to retain them for debugging.

Production-shaped collector deployment

• Exercise a container function with the collector as a workload sidecar.
• Exercise a Helm function with the collector on the generated utils pod.
• Generate both workloads through the shared translation library rather than handwritten collector manifests.
• Supply representative synthetic function launch specifications and equivalent translation inputs.
• Validate collector placement, image, resources, ports, volumes, environment, probes, and configuration before each run.
• Fail clearly when translator output no longer produces the expected test shape.

Collector images

• Support building the collector from the current local source for k3d testing.
• Support an explicit collector image reference.
• Import locally built images into k3d.
• Require a registry-accessible image for remote clusters.
• Record the exact image and source revision in results.

Telemetry generation

• Run telemetry load generation inside the target cluster.
• Use an existing open source OTLP load generator where practical.
• Make load rate, duration, concurrency, batching, and payload characteristics configurable.
• Run logs and metrics independently and as a concurrent mixed workload.
• Cover normal-sized and oversized log records, including the collector log-chunking path.
• Cover representative metric types, normal cardinality, high cardinality, and parallel generation where needed.

Execution profiles

• Provide a short dev profile for wiring validation and local iteration.
• Provide a repeatable baseline profile with warmup, a defined measurement window, and repeated measurements.
• Document profile defaults while keeping durations, rates, repetitions, and payload sizes configurable.

Initial telemetry destination

• Provide an in-cluster OTLP sink that accepts, counts, and discards telemetry.
• Expose enough sink information to reconcile collector output with received records.
• Do not require an external observability backend.
• Leave room for supported real-backend targets in a later version.

Measurements

Collect enough information to characterize normal operation and saturation behavior:

• Generator attempted, successfully exported, and failed record counts.
• Collector accepted, refused, processed, sent, retried, and dropped record counts where exposed.
• Records received by the OTLP sink.
• Collector CPU and memory behavior.
• Queue utilization and backpressure indicators.
• Pod restarts, out-of-memory events, readiness failures, and export errors.
• Processor and exporter latency metrics where exposed.
• Achieved throughput at fixed input rates.
• Maximum sustainable throughput before drops, persistent queue growth, or resource exhaustion.

True generator-to-sink latency correlation is not required in the initial version.

Results and documentation

• Produce a concise human-readable summary and structured results for later comparison or automation.
• Include test parameters, deployment shape, profile, collector image and version, source revision, cluster metadata, timestamps, and collected measurements.
• Save raw measurements needed to investigate a result.
• Exclude credentials and secret material from output artifacts.
• Document prerequisites, profiles, deployment-shape selection, image selection, load customization, report interpretation, environmental variance, cleanup, and retained-resource debugging.

Definition of Done

• The four linked Task issues are complete.
• Both container and Helm deployment shapes can be exercised in k3d and remote modes.
• The suite produces reproducible logs, metrics, measurements, and result artifacts.
• Operator documentation covers setup, execution, interpretation, and cleanup.

Out of Scope

• CI pass/fail performance thresholds.
• Centralized historical baseline storage.
• Automatic comparison against previous results.
• True end-to-end latency correlation.
• Remote cluster provisioning.
• Mandatory external observability backend integration.
• Full control-plane installation and reconciliation.

Tasks

• byoo-perf-suite: Add scaffolding, translation-driven rendering, and validation #417
• byoo-perf-suite: Add collector deployment and image handling for k3d and remote modes #418
• byoo-perf-suite: Add load generation, OTLP sink, and measurements #419
• byoo-perf-suite: Add reporting and documentation #420

Resources

• src/compute-plane-services/byoo-otel-collector/
• src/libraries/go/lib/pkg/icms-translate/

---

By submitting this issue, you acknowledge that you are an assigned member of the NVCF development team and agree to follow our code of conduct and our contributing guidelines.