feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders

[rhuss]

User Story

As a security or compliance reviewer investigating a policy violation, I want each OCSF event to carry the ID of the trace that produced it, so that I can jump from the event in my log aggregator straight to the supervisor's trace instead of matching sandbox IDs and timestamps by hand.

Problem Statement

Open Cybersecurity Schema Framework (OCSF) security events and OpenTelemetry (OTel) traces exist in separate systems with no connection between them. A security reviewer filters the OCSF log aggregator (Loki, Splunk) for DENY events in the last hour and finds a network deny for api.suspicious.com from sandbox sb-abc123. To see how the supervisor handled that connection (which policy matched, what L7 enforcement and middleware ran), they search Jaeger for the sandbox ID and hope the timestamps line up. Nothing links the security event to its trace.

Today only the gateway emits OTel spans. Once #3977 lands, the supervisor does too, and most OCSF events and OTel spans then come from the same code paths. The OCSF builders have no way to record the active trace context.

Impact / Why This Matters

Every deny investigation needs manual correlation across two systems. The workaround is searching the trace backend by sandbox ID within a time window. That breaks down quickly: with #3977 every egress connection starts its own trace, so a busy sandbox produces many traces per second and the time window rarely identifies one. SIEM pipelines also have no join key to automate the correlation.

Proposed Design

Record the active trace context in OCSF events using the OCSF 1.8 Trace profile. When a sampled OTel span is active at emission time, the event carries trace.uid and lists trace in metadata.profiles. Without one, the trace object is omitted. The reviewer pastes trace.uid into Jaeger or Tempo and lands on the supervisor trace that produced the event.

Representation

OCSF 1.8 has no top-level trace_id or span_id attributes. It defines a Trace profile whose trace object carries the W3C trace ID in trace.uid, plus an optional span object:

{"class_uid": 4001, "activity_id": 1, "metadata": {"profiles": ["security_control", "network_proxy", "container", "host", "trace"], ...}, "trace": {"uid": "0af7651916cd43dd8448eb211c80319c"}, ...}

The first version populates trace.uid only. The 1.8 span object requires start_time and end_time, which are unknown while the span is still open at emission time. Whether and how to fill trace.span is settled during implementation against the vendored-schema validation tests. The Trace profile schema files get vendored next to the existing ai_operation profile.

Layering

openshell-ocsf stays independent of OpenTelemetry:

1. openshell-ocsf adds a plain-data trace correlation type and a builder setter that renders the trace object and adds the profile.
2. openshell-otel provides a function that extracts the active OTel context and returns it only when the span context is valid and sampled. An unsampled trace ID points to nothing in the trace backend.
3. Binaries that emit OCSF events register this extractor with openshell-ocsf at startup. ocsf_emit! fills in the trace context when the builder has not set one, so individual call sites need no OTel dependency. An explicit setter call overrides the automatic value.

Egress deny spans

The supervisor egress spans from #3977 are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level. Without a change, network denies get no trace ID at default settings. The proposal is an INFO-level span around deny decisions only. Denies are rare and are what reviewers investigate, while allowed connections stay at DEBUG to keep span volume down.

Out of scope: agent trace context

Each egress connection starts its own root trace, so the linked trace shows the supervisor's handling of the connection, not the agent's activity. Linking a deny to the agent's own trace, through the workload's traceparent header relayed via #3196, is a follow-up. The workload controls that header, so it goes in a separate, clearly labelled field and never in the trace object.

Scope

• crates/openshell-ocsf/: trace correlation type, builder setter, trace serialization, metadata.profiles entry, vendored Trace profile schema, emission hook
• crates/openshell-otel/: sampled trace context extraction
• crates/openshell-supervisor-network/: INFO-level span around deny decisions
• Supervisor and gateway binaries: register the extractor at startup
• No breaking changes (the trace object is optional)

Dependencies

• feat(supervisor): export OTLP traces from sandbox supervisors #3977 (supervisor OTLP span export) must land first.

Acceptance Criteria

• OCSF events emitted inside a sampled OTel span include trace.uid (32 lowercase hex characters) and list trace in metadata.profiles.
• OCSF events emitted without an active sampled span omit the trace object and do not list the profile.
• At the default supervisor log level, OCSF network and HTTP deny events carry a trace.uid that resolves in the configured trace backend.
• openshell-ocsf has no OpenTelemetry dependency.
• The Trace profile schema is vendored, and the schema validation tests cover events with and without trace.
• The published observability docs describe the trace field and how to follow it to the trace backend.

Alternatives Considered

Top-level trace_id / span_id fields: Easy to query, but outside the OCSF 1.8 schema. Consumers that validate against OCSF treat them as unknown attributes. The standard Trace profile covers the same need.

The unmapped mechanism: Schema-valid, but unmapped is meant for source data that has no OCSF attribute. Trace context has one, and downstream tooling won't look for it in unmapped.

Per-site .trace_context() calls: Explicit, but every emitting crate would need an OTel dependency, and adoption would drift across the call sites. The emission hook covers every event, and the explicit setter stays available where the current span is the wrong source.

Enrichment in the JSONL layer: Works, but puts the trace context into one output format instead of the event itself, so the shorthand format and any future sink would miss it.

Raising all egress spans to INFO: Gives every OCSF network event a trace ID, but exports one trace per outbound connection at default settings. Scoping the INFO span to denies covers the investigation case at a fraction of the volume.

Agent Investigation

• OCSF builders live in crates/openshell-ocsf/src/builders/. Each builder declares its profiles through ctx.metadata(&[...]), and api_activity.rs already adds ai_operation conditionally.
• The vendored schemas cover OCSF 1.8.0 but include only the ai_operation profile. The Trace profile is available from the OCSF schema server.
• OCSF events are emitted via ocsf_emit!(), which stores the event in a thread-local and emits via tracing::info!(). The shorthand and JSONL layers extract it from there.
• openshell-otel owns OTel context handling via tracing_opentelemetry::OpenTelemetrySpanExt. The supervisor crates have no OTel dependency on main.
• ocsf_emit! call sites exist in openshell-sandbox, openshell-server, openshell-supervisor, openshell-supervisor-network, and openshell-supervisor-process.
• feat(supervisor): export OTLP traces from sandbox supervisors #3977 (closes feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508) adds supervisor span export. Its egress spans (supervisor.egress.connect, authorize, resolve, dial) are DEBUG, and the OTLP filter is info,openshell=<level> with a floor of INFO.

Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission), #3977 (supervisor OTLP span export), #3196 (OTLP relay), #2507 (Gateway OTel export surface)

[kvnloo]

I checked this against the current crate boundary and the vendored OCSF schema.

One wrinkle: the OCSF 1.8 base schema does not define top-level trace_id / span_id fields for these events. It does provide the schema-bounded unmapped escape hatch.

There is also a dependency-boundary question:

• openshell-ocsf currently owns OCSF event construction and depends on tracing;
• openshell-otel already owns OpenTelemetry context extraction through tracing_opentelemetry::OpenTelemetrySpanExt.

I would prefer not to make the OCSF crate itself responsible for discovering the active OpenTelemetry context.

A narrow design could be:

1. let the OCSF builders accept optional trace/span correlation values;
2. represent them through the schema-supported extension/unmapped path;
3. have the OTEL layer extract the active context and pass those values into the event builder.

That keeps OCSF serialization independent of the OpenTelemetry implementation.

If top-level trace_id / span_id fields are intended as an OpenShell-specific extension instead, I think that should be explicit in the contract because it is outside the vendored base schema.

Which representation do you prefer?

[rhuss]

Thanks for checking this against the schema, @kvnloo. You're right on both counts. The dependency point also exposes a bug in my original proposal: tracing::Span::current() only gives you process-local tracing IDs, not W3C trace and span IDs.

Representation. I'd go with neither top-level fields nor unmapped. OCSF 1.8 defines a Trace profile for this case, with a trace object that holds the W3C trace ID in trace.uid. We only vendor the ai_operation profile today, so it's easy to miss. An event would look like this:

{
  "metadata": { "profiles": ["security_control", "network_proxy", "container", "host", "trace"] },
  "trace": { "uid": "0af7651916cd43dd8448eb211c80319c" }
}

One catch: the 1.8 span object requires start_time and end_time, and we emit the OCSF event while the span is still open. So I'd ship trace.uid first, since that's what a security reviewer pastes into Tempo or Jaeger anyway, and settle trace.span in the PR against the vendored-schema validation tests.

Layering. Agreed, openshell-ocsf shouldn't know about OpenTelemetry. What I have in mind:

1. openshell-ocsf gets a plain-data correlation type and a builder setter that renders the trace object and adds the profile.
2. openshell-otel extracts the active context and returns it only when the span context is valid and sampled. An unsampled trace ID sends the reviewer to an empty search result, which is worse than no ID.
3. Binaries that emit OCSF events register that extractor with openshell-ocsf at startup, and ocsf_emit! fills in the context when the builder hasn't set one. The emitting crates don't each need an OTel dependency, and an explicit setter still wins where the current span is the wrong source.

Prerequisite. I also got something wrong in the issue: on main, the supervisor crates have no OpenTelemetry instrumentation yet. #3977 adds it, so this should build on that PR. Even with #3977 there are two gaps for the network-deny case:

• The egress spans are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level, so at default settings a deny would still carry no exported trace ID. I'd lean toward an INFO span around deny decisions only. Denies are rare and they're what a security reviewer investigates, while allowed connections stay at DEBUG to keep the span volume down.
• Each egress connection starts its own root trace. That trace shows how the supervisor handled the connection, not what the agent was doing. The agent's side lives in the workload's own traceparent header (relayed via feat(observability): supervisor OTLP telemetry relay #3196). The workload controls that header, so it needs a separate, clearly labelled field and must not end up in the trace object. I'd handle that as a follow-up.

I'll update the issue body to reflect this. Are you planning to pick up the implementation? If so, building on top of #3977 makes sense.