OTLP browser telemetry export: follow-up improvements (KERNEL-1596)

[archandatta]

tldr

Follow-up improvements to the in-VM OTLP browser-telemetry exporter (KERNEL-1596), post #308. Independent, reviewable commits. Off-by-default export is preserved; nothing changes for existing sessions unless the new toggle/knobs are set.

What changed

1. Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE (2048), BTEL_OTLP_EXPORT_INTERVAL (1s), BTEL_OTLP_EXPORT_TIMEOUT (30s). Defaults match the OTel SDK's, so behavior is unchanged unless set; validated as > 0.
2. Per-session export toggle — new export.otlp.enabled on the telemetry API. Export is off by default and gated on the toggle, independent of whether an export destination is configured, so an always-injected relay does not generate work that gets dropped. The writer moves under a runtime-toggleable controller; toggling is best-effort (never fails a telemetry apply).
3. Drop / failure / exported metrics — queue-overflow drops (previously log-only) plus export failures and exported counts as Prometheus counters (kernel_otlp_records_dropped_total, kernel_otlp_export_failures_total, kernel_otlp_records_exported_total), owned above the writer so they stay monotonic across export toggles.
4. Crash/OOM severity — service_crashed and system_oom_kill now map to ERROR in the OTLP converter (were INFO), so consumers can alert on them.
5. Renderer-crash event — detect Inspector.targetCrashed (an "Aw, Snap!" renderer crash that leaves the browser process alive, so no service_crashed fires) and emit a page_crashed event, mapped to ERROR. Categorized as page (produced by the CDP monitor, captured only while the page category is enabled).

Live test results

Verified full-stack against the headless image + a real dockerized OpenTelemetry collector, driving the real telemetry API and a real Chromium over CDP:

• Batch knobs: custom BTEL_OTLP_* values surfaced in the boot config (otlp_max_queue_size=512 otlp_export_interval=500ms otlp_export_timeout=30s); export shows available, not started, at boot.
• Export toggle: GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it. With export off but page/network capture on, exported_total froze (10 → 10) and the collector received nothing new; re-enabling resumed export (10 → 28), confirming the controller rebuilds the one-shot writer.
• Renderer crash: a real chrome://crash produced a page_crashed record at the collector with category=page, url=chrome://crash/, and SeverityText: ERROR.
• Converter fidelity: network_response carried promoted attributes (http.request.method, url.full, http.response.status_code=200); screenshot/monitor categories never reached the collector.
• Metrics: /metrics served kernel_otlp_records_dropped_total 0, kernel_otlp_export_failures_total 0, kernel_otlp_records_exported_total 28.

🤖 Generated with Claude Code

---

Note

Medium Risk
Changes default OTLP behavior (no export until API toggle) and extends the telemetry API contract; export reconciliation is best-effort and should not break capture, but operators relying on always-on export must enable the new flag.

Overview
Follow-up on in-VM OTLP browser telemetry: export is gated per session via export.otlp.enabled on PUT/PATCH /telemetry (off by default even when a relay endpoint is provisioned). The API wires an OTLPExportController through reconcileExport / stopTelemetryState; boot only constructs the sink and stops it after HTTP drain, mirroring S2.

Operational visibility: BTEL_OTLP_MAX_QUEUE_SIZE, BTEL_OTLP_EXPORT_INTERVAL, and BTEL_OTLP_EXPORT_TIMEOUT tune the OTel batch processor; Prometheus counters (kernel_otlp_records_*) track drops, export failures, and successes via a drop-counting slog handler on the OTel global logger.

Events: CDP monitor enables Inspector and emits page_crashed on renderer crashes; OTLP severity maps page_crashed, service_crashed, and system_oom_kill to ERROR. Re-enabled export reads from the event stream tail so the ring is not replayed.

^{Reviewed by Cursor Bugbot for commit be7dc1d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[cursor]

Export enable can skip in-flight events

Medium Severity

Start snapshots es.Seq() then builds a tail reader, but Publish increments seq before ring.publish (and may spend time in truncateIfNeeded). An event whose seq is observed during that window is skipped forever by the new reader, even if it lands in the ring after export is enabled. Under concurrent capture this can drop the first events after an export toggle-on.

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[cursor]

Drop counter wrapper loses attrs

Medium Severity

dropCountingHandler embeds slog.Handler and only overrides Handle, so promoted WithAttrs / WithGroup return the unwrapped base handler. Any logr/OTel path that names or attaches values before logging dropped log records bypasses the counter, leaving kernel_otlp_records_dropped_total stuck at 0 despite real queue overflow.

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[cursor]

Stale URL on page crash

Medium Severity

page_crashed fills url from m.sessions attach-time target info, which is never updated on navigation. Other page events use the computed nav context’s current URL. After the first navigation, crash events report the wrong page URL even though the OpenAPI text describes the URL at crash time.

Additional Locations (1)

• server/lib/cdpmonitor/handlers.go#L730-L735

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[cursor]

Mutex held during export drain

Medium Severity

Disabling OTLP export calls Stop while monitorMu is held, and OTLPExportController.Stop also holds its own mutex for the full drain (up to otlpStopTimeout, 5s). That stalls all telemetry GET/PUT/PATCH handlers and ApiService.Shutdown for the drain window.

Additional Locations (1)

• server/lib/events/otlpstorage.go#L343-L354

 

^{Reviewed by Cursor Bugbot for commit be7dc1d. Configure here.}

[Sayan-]

Approve — reviewed all 11 commits (correctness + adversarial), ran the affected Go test packages, and rebuilt the headless image to re-run the live test plan end-to-end against a real dockerized OTel collector. Everything checks out.

Commit-by-commit

• Externalize batch knobs — BTEL_OTLP_MAX_QUEUE_SIZE/EXPORT_INTERVAL/EXPORT_TIMEOUT, validated > 0, defaults match the OTel SDK so behavior is unchanged unless set. Applied only when > 0 in newOTLPStorage. Good.
• Per-session export toggle — export is off by default and gated on export.otlp.enabled, independent of whether an endpoint is provisioned; writer lifecycle moves under OTLPExportController (one-shot writer rebuilt per enable). Rollback-on-failure, PATCH-preserves / PUT-replaces semantics, and the no-op short-circuit all trace correctly.
• Drop/failure/exported metrics — verified the full chain against vendored SDK v0.20.0: global.Warn("dropped log records", "dropped", d uint64) → V(1).Info → logr→slog level Info-1 → the Info-1 handler threshold in main.go. attrUint64 handles all three type boxings. Counters owned above the writer so they stay monotonic across toggles.
• Crash/OOM → ERROR — service_crashed, system_oom_kill, page_crashed map to ERROR in the converter.
• Renderer-crash event — Inspector.enable added to enableDomains; Inspector.targetCrashed → page_crashed (category page). Untracked-session path falls back to target_type: other (valid enum) rather than "", with a regression test. Category gating confirmed: the monitor publishes via telemetrySession.Publish, which drops page events unless the page category is active.
• Shutdown ordering (Bugbot) — OTLP stop moved to main after the servers drain (mirrors s2Writer); Stop's ctx-exhausted branch closes the provider on a detached 2s budget so a hung flush can't orphan batch-export goroutines. Correct.

Tests

go test ./lib/events/ ./lib/cdpmonitor/ ./lib/metrics/ ./cmd/api/api/ ./cmd/config/ — all green. TestOTLPMetrics_CountsRealDrops forces a real batch-queue overflow through the SDK global logger, so a future SDK message rename fails CI instead of silently zeroing the metric — nice guard.

Live test plan (rebuilt headless image + real OTel collector)

Built the image and ran it against a dockerized otel-collector-contrib + an nginx target, driving the real telemetry API and real Chromium 148 over CDP (custom knobs 512 / 500ms / 30s). All PR claims reproduced:

• Boot config surfaced the knobs; OTLP export available (not started).
• GET /telemetry → 404 at boot; PUT with export.otlp.enabled:true → 201 echoing it.
• Real chrome://crash → page_crashed at collector with SeverityText: ERROR / SeverityNumber: Error(17), source.event=Inspector.targetCrashed, category=page.
• Converter fidelity: network_response carried http.request.method, url.full, http.response.status_code=Int(200); zero screenshot/monitor records reached the collector.
• Toggle: with export off, a full page navigation produced zero new exported records (exported_total frozen 22→22); re-enabling rebuilt the writer and resumed (22→54).
• /metrics: kernel_otlp_records_dropped_total 0, export_failures_total 0, records_exported_total climbing.

Non-blocking notes (no changes required)

1. reconcileExport holds monitorMu for up to otlpStopTimeout (5s) during a toggle-off drain; if the relay is unreachable this can briefly head-of-line-block concurrent GET/PUT/PATCH /telemetry. Latency only, not correctness — consider draining outside the lock if toggling gets frequent.
2. BTEL_OTLP_MAX_QUEUE_SIZE is only validated > 0 while the export batch size is fixed at 200; setting it below 200 won't crash but interacts oddly with the constant. Consider validating >= 200 or a doc note.
3. PUT with an omitted export block disables export (defaults false) — correct PUT-replace semantics and consistent with how omitted categories turn off, but a sharp edge for consumers; PATCH correctly preserves. Worth a sentence in the openapi description.

Nice, well-tested change. LGTM.