feat(integrations): Splunk data source + anomaly forwarding (#99) - #145
Draft
Jovonni wants to merge 1 commit into
Draft
feat(integrations): Splunk data source + anomaly forwarding (#99)#145Jovonni wants to merge 1 commit into
Jovonni wants to merge 1 commit into
Conversation
Adds a bidirectional Splunk integration, reusing the existing connector / data-source / integration-settings patterns: - SplunkConnector (core/integrations): REST export search (input) + HTTP Event Collector (output), bearer or basic auth, env + integration_settings config - model runner: 'splunk' data_source branch runs an SPL search and returns a DataFrame (self-contained, mirrors the elasticsearch branch) - models train/execute: splunk_search / splunk_index params + dispatch - orchestrator: best-effort forward of each run's anomalies to Splunk HEC when enabled (forward_anomalies), never blocking detection - settings: register 'splunk', mask token/hec_token/password, /test probe - settings UI: Splunk card under Data Engines - docs/SPLUNK.md - dedicated test subsuite (26 unit tests, fully mocked, runs in the CI unit job) Closes #99
There was a problem hiding this comment.
Pull request overview
This PR adds a bidirectional Splunk integration to OpenUBA: querying Splunk as a model data source (via the REST export endpoint) and optionally forwarding detected anomalies back to Splunk (via HEC), along with settings UI/API wiring and a dedicated unit test suite.
Changes:
- Added a
SplunkConnector(REST export search + HEC output) and registered it in integrations and settings test-probe/masking/whitelist plumbing. - Extended the model runner and model train/execute API to support a
splunkdata source withsplunk_search(and exposedsplunk_index). - Added orchestrator-side best-effort anomaly forwarding to Splunk HEC plus unit tests and user documentation.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| interface/src/components/settings/settings-tabs.tsx | Adds a Splunk integration card/fields to the settings UI. |
| docs/SPLUNK.md | Documents Splunk configuration, SPL search usage, and anomaly forwarding. |
| docker/model-runner/runner.py | Adds a splunk branch in _load_data_for_model to run Splunk export searches and build a DataFrame. |
| core/integrations/splunk.py | Introduces SplunkConnector for search + HEC event/anomaly forwarding + connectivity probe. |
| core/integrations/init.py | Exposes SplunkConnector from the integrations package. |
| core/api_routers/settings.py | Whitelists splunk, adds /test dispatch to _test_splunk, and masks Splunk secrets. |
| core/api_routers/models.py | Adds splunk_search/splunk_index request parameters and input-data dispatch for train/execute. |
| core/services/model_orchestrator.py | Adds best-effort forwarding of anomalies to Splunk HEC after inference persistence (Docker mode only as written). |
| core/tests/test_services/test_splunk_connector.py | Unit tests for connector normalization, auth, search parsing, HEC output, and connectivity probe. |
| core/tests/test_services/test_splunk_wiring.py | Unit tests for settings whitelist/masking/test probe and orchestrator forwarding gating. |
Suppressed comments (1)
core/api_routers/models.py:488
- Same issue as the train endpoint:
splunk_indexis forwarded intoinput_databut ignored by the model-runner Splunk loader, and there is no wiring here to provide the required Splunk host/auth into the runner container for Splunk searches.
elif _data_source == "splunk" and _splunk_search:
input_data["splunk_search"] = _splunk_search
if _splunk_index:
input_data["splunk_index"] = _splunk_index
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+288
to
+292
| async def _test_splunk(config: dict) -> dict: | ||
| '''verify Splunk connectivity via the shared SplunkConnector''' | ||
| from core.integrations.splunk import SplunkConnector | ||
| return SplunkConnector.from_config(config).test_connection() | ||
|
|
Comment on lines
175
to
+178
| db.commit() | ||
|
|
||
| # optional: forward anomalies to Splunk (HEC) if configured | ||
| self._forward_anomalies_to_splunk(result["anomalies"]) |
Comment on lines
+373
to
+376
| elif _data_source == "splunk" and _splunk_search: | ||
| input_data["splunk_search"] = _splunk_search | ||
| if _splunk_index: | ||
| input_data["splunk_index"] = _splunk_index |
Comment on lines
+40
to
+44
| -d '{ | ||
| "data_source": "splunk", | ||
| "splunk_search": "index=proxy sourcetype=access_combined", | ||
| "splunk_index": "proxy" | ||
| }' |
Comment on lines
+412
to
+415
| headers = {"Content-Type": "application/x-www-form-urlencoded"} | ||
| if token: | ||
| headers["Authorization"] = f"Bearer {token}" | ||
| auth = (username, password) if (not token and username) else None |
Comment on lines
+198
to
+204
| def send_anomalies(self, anomalies: List[Dict[str, Any]], index: Optional[str] = None) -> int: | ||
| '''forward a batch of anomalies; returns the count successfully sent''' | ||
| sent = 0 | ||
| for a in anomalies: | ||
| if self.send_anomaly(a, index=index): | ||
| sent += 1 | ||
| return sent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Integrates Splunk in both directions (requested in #99):
How it reuses existing infrastructure
SplunkConnector(core/integrations/splunk.py) sits alongsideElasticsearchConnector/SparkConnector— REST export search for input, HTTP Event Collector for output, bearer or basic auth. Config viaintegration_settings(from_config) orSPLUNK_*env.splunkbranch in_load_data_for_model, mirroring the elasticsearch branch (self-containedrequests, returns a DataFrame with numeric coercion).splunk_search/splunk_indexparams + a dispatch branch, exactly like the existingspark/elasticsearch/source_groupsources.forward_anomaliesis enabled — best-effort, so a Splunk outage never affects detection or persistence.splunkregistered in the whitelist, secrets (token/hec_token/password) masked,/testconnectivity probe, and a Splunk card in the Data Engines settings panel.docs/SPLUNK.md.Tests (dedicated subsuite, CI-wired)
core/tests/test_services/test_splunk_connector.py+test_splunk_wiring.py— 26 unit tests, all mocked (no live Splunk): SPL normalization, search parsing + bearer/basic auth, error handling, HEC output + auth header, batch send counts, connectivity probe, settings whitelist/masking/_test_splunk, and orchestrator forwarding gating (enabled/disabled/flag-off/error-safe). Placed undertest_services/so they run in the CI unit job (the-kfilter excludes anything undertest_integrations/).Local: 26/26 green, runner compiles,
tsc/next lint/next buildall clean.Follow-up (not in this PR)
Splunk-as-a-physical-source inside
source_groups(so a source group can mix Splunk with other sources) is a natural next step.Closes #99