Skip to content

feat(ocsf): configurable schema version for SIEM backward compatibility - #2717

Open
zanetworker wants to merge 1 commit into
NVIDIA:mainfrom
zanetworker:feat/ocsf-configurable-schema-version
Open

feat(ocsf): configurable schema version for SIEM backward compatibility#2717
zanetworker wants to merge 1 commit into
NVIDIA:mainfrom
zanetworker:feat/ocsf-configurable-schema-version

Conversation

@zanetworker

Copy link
Copy Markdown
Contributor

Summary

Add a gateway-configurable OCSF schema version target that downgrades JSONL output for SIEMs that only support older schema versions. AWS Security Lake requires v1.1.0, Splunk CIM Add-On targets v1.1-v1.3, CrowdStrike FDR uses v1.5.0. OpenShell's internal model stays at the current version; only the JSONL serialization is transformed.

Related Issue

Closes #2662

Approach

A downgrade filter in the JSONL layer runs after event serialization and before writing. Based on a field-by-field diff across v1.1.0, v1.3.0, and v1.7.0 (analysis), the filter is small:

  1. Strip profile-gated fields: ai_model, container, observation_point_id
  2. Remove unknown profiles from metadata.profiles: ai_operation, container
  3. Rewrite metadata.version to the target version

No field renames needed. The core event structure (class UIDs, activity IDs, HTTP/network fields) is identical across all versions. The differences are all profile-gated additions.

The shorthand log output is unaffected (human-readable, not consumed by SIEMs).

Gateway Configuration

[openshell.gateway]
ocsf_schema_version = "1.1"

The setting flows to sandboxes via the settings bundle (same mechanism as ocsf_json_enabled). Empty or unset means no downgrade.

Changes

openshell-ocsf crate:

  • format/downgrade.rsdowngrade_event() function that strips fields and profiles based on target version
  • tracing_layers/jsonl_layer.rswith_target_version() method; applies downgrade before writing

openshell-sandbox crate:

  • main.rs — creates shared ocsf_schema_version mutex, passes to JSONL layer
  • lib.rsapply_ocsf_schema_version_setting() reads the gateway setting and updates the shared version

SIEM Compatibility Matrix

SIEM OCSF Version Downgrade Needed
AWS Security Lake v1.1.0 Yes
Splunk CIM Add-On v1.1-v1.3 Yes
CrowdStrike FDR v1.5.0 Yes (strips v1.7+ fields)
Datadog Cloud SIEM v1.5.0 (selectable) Depends on config

Testing

  • cargo test -p openshell-ocsf — 134 passed (7 new downgrade tests)
  • cargo clippy --all-targets -- -D warnings — clean
  • cargo check -p openshell-sandbox — compiles with settings wiring

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Unit tests added

@copy-pr-bot

copy-pr-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@zanetworker
zanetworker force-pushed the feat/ocsf-configurable-schema-version branch from 4697da3 to 34073ea Compare August 12, 2026 11:32
Add a gateway-configurable OCSF schema version target that downgrades
JSONL output for SIEMs that only support older schema versions (v1.1 for
AWS Security Lake, v1.3 for Splunk CIM).

The downgrade filter runs after event serialization and before writing:
- Strips profile-gated fields (ai_model, container, observation_point_id)
- Removes unknown profiles from metadata.profiles
- Rewrites metadata.version to the target version

No field renames needed; the core event structure is identical across
v1.1, v1.3, and v1.7. The differences are all profile-gated additions.

The gateway setting ocsf_schema_version flows to sandboxes via the
settings bundle (same path as ocsf_json_enabled). Empty or unset means
no downgrade (emit at current version).

Closes NVIDIA#2662

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@zanetworker
zanetworker force-pushed the feat/ocsf-configurable-schema-version branch from 34073ea to 8fdaeec Compare August 12, 2026 12:17
@zanetworker

Copy link
Copy Markdown
Contributor Author

E2E Verification: Splunk CIM mapping with v1.1 downgraded events

Tested the full pipeline on a live cluster with Splunk Enterprise:

  1. Gateway setting: ocsf_schema_version = "1.1"
  2. Sandbox emits events internally at v1.7, JSONL layer downgrades to v1.1
  3. Events sent to Splunk HEC with sourcetype=ocsf
  4. OCSF-CIM Add-On maps events to standard CIM data models

Splunk CIM query: sourcetype=ocsf tag=network tag=communicate | table _time dest dvc vendor_product metadata.version

Result: 13 events mapped to CIM Network Traffic data model with metadata.version = 1.1. CIM fields (dest, dvc, vendor_product) populated automatically via the OCSF-CIM Add-On's eventtypes and field aliases. No custom dashboards or SPL required.

Splunk CIM v1.1 downgrade

This proves the "plug OpenShell into your existing SIEM and it just works" story for customers running Splunk with the OCSF-CIM Add-On on v1.1 schemas.

@zanetworker

Copy link
Copy Markdown
Contributor Author

Related: #2664 adds AI inference events via the ai_operation profile on ApiActivity [6003] and bumps the schema to v1.8.0. The downgrade filter in this PR strips those profile fields when targeting v1.1/v1.3 for older SIEMs. The two PRs are independent and can be merged in either order.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Configurable OCSF schema version for SIEM compatibility (v1.1/v1.3)

1 participant