feat(data): dataset ingestion + detection evaluation (#35) [WIP] by Jovonni · Pull Request #146 · GACWR/OpenUBA

Jovonni · 2026-08-22T20:27:49Z

Draft / WIP for #35 (Data Ingestion & Testing). Building this on OpenUBA's existing rails — deliberately not inventing a parallel workflow.

Background

#35 asked for sample data + a way to test detections. The thread's real blocker (anupamme): the data has no labels, so you can't evaluate whether a model works. This delivers the missing evaluation capability and makes additional datasets first-class.

Design was hardened by auditing the code first, which corrected several early assumptions: models featurize raw logs themselves (no pre-aggregation), the canonical run path is ES/Spark (not local_csv), ground truth should reuse UserFeedback, and metrics belong in ExperimentRun.metrics — so this reuses those instead of rebuilding them.

Phase A (this PR so far) — all tested

Detection scorer (core/services/detection_eval.py): pure, dependency-light entity-level precision/recall/F1/FPR (+ per-scenario recall), shaped for ExperimentRun.metrics.
Run-scoped anomaly reads: AnomalyRepository.list_all(run_id=…), a run_id param on GET /api/v1/anomalies, and query_anomalies(run_id=) in the SDK (the read evaluation needs).
Evaluation endpoint + SDK: POST /api/v1/evaluate/run (reads a run's anomalies, scores them) and openuba.evaluate_run(run_id, malicious_entities, …).
Ingest any dataset: replaced the hard toy_1-only whitelist in ingest_all_datasets with a "looks like a dataset" check — additional captures under test_datasets/ now fan out to their Spark tables + ES indices with no code change (toy_1 unchanged).
Docs: docs/DATASETS_AND_EVALUATION.md.

Tests (18, green locally; run in CI)

unit (no infra): 10 scorer + 2 dataset-scan
API/repo (testcontainers Postgres): 2 evaluation endpoint + 1 run_id filter
SDK: 3

Phase B (follow-up in this branch)

A small real-format labeled capture under test_datasets/ with injected known-bad entities + a ground_truth.json.
Seed Dataset rows (none are seeded today) so shipped datasets show on /datasets.
An e2e test: ingest → train/infer via ES → read anomalies by run → evaluate → assert recall on known-bad entities ≥ bar; extend the JIT matrix.
A starter notebook (extends 09_model_pipeline.ipynb with an eval cell) and a CLI dataset group.

Also fixes two real bugs found while mapping: jobs.py local_csv missing file_name, and the dead source_group LocalPandasLoader import (to be addressed in Phase B).

Closes #35

) First slice of the Data Ingestion & Testing work, on existing rails: - core/services/detection_eval.py: pure, dependency-light entity-level scoring (precision/recall/F1/FPR + per-scenario recall) shaped for ExperimentRun.metrics - anomalies read-by-run: AnomalyRepository.list_all(run_id=...), a run_id query param on GET /api/v1/anomalies, and query_anomalies(run_id=) in the SDK — the read path evaluation needs to score a single run's output - tests: 10 unit tests for the scorer (no infra) + a repo test for the run_id filter

Replace the hard toy_1-only whitelist in ingest_all_datasets with a 'looks like a dataset' check (directory with >=1 log-type subdir). toy_1 behavior is unchanged; additional datasets now fan out to their <name>_<logtype> Spark tables + openuba-<logtype>-<name> ES indices with no code change. Junk/hidden dirs and stray files are still skipped. Tests: dataset-scan unit tests (temp tree, mocked ingest — no infra).

- POST /api/v1/evaluate/run: reads a run's anomalies (by run_id) and scores them with the shared core.services.detection_eval scorer; returns JSONB- friendly metrics ready for ExperimentRun.metrics - SDK: openuba.evaluate_run(run_id, malicious_entities, ...) + query_anomalies now accepts run_id - tests: API test (seed run+anomalies, POST, check metrics) + 3 SDK unit tests

- inject an admin caller (override get_current_user) so the evaluation API test exercises the real require_permission path and returns 200 - pass the required slug to ModelRepository.create in the new tests - docs/DATASETS_AND_EVALUATION.md: adding datasets, ground truth, and evaluating a run (precision/recall/f1) via the SDK, reusing experiments

This branch touches sdk/, which triggers the workspace image build; it fails generating pyspark metadata without modern build tooling. Same fix already validated on the alerts branch.

Copilot

Pull request overview

Adds end-to-end support for evaluating a model run’s anomaly detections against ground-truth entity labels (issue #35), while expanding dataset ingestion to automatically pick up additional captures under test_datasets/ without code changes.

Changes:

Introduces a dependency-light, entity-level detection scoring service (precision/recall/F1/FPR + per-scenario recall) and corresponding unit tests.
Adds run-scoped anomaly listing across repository/API/SDK, plus a new /api/v1/evaluate/run endpoint and SDK surface to score a run.
Broadens dataset ingestion discovery from a toy_1 whitelist to “looks like a dataset” directory scanning, with docs and tests.

Reviewed changes

Copilot reviewed 15 out of 16 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/tests/test_evaluate.py	Adds SDK tests for `evaluate_run` payload + `run_id` anomaly scoping.
sdk/src/openuba/client.py	Extends anomaly query with `run_id` and adds `evaluate_run` SDK method.
sdk/src/openuba/init.py	Exposes `evaluate_run` and `run_id`-scoped `query_anomalies` at module level.
docs/DATASETS_AND_EVALUATION.md	Documents dataset layout, ingestion behavior, and run evaluation workflow.
docker/workspace/Dockerfile	Upgrades build tooling to support sdist/isolated-build installs.
core/tests/test_services/test_detection_eval.py	Unit tests for entity-level detection evaluation math.
core/tests/test_services/test_data_ingestion_scan.py	Tests dataset directory scanning behavior for ingestion fan-out.
core/tests/test_repositories.py	Adds repository test for `AnomalyRepository.list_all(run_id=...)`.
core/tests/test_api_routers/test_evaluation.py	API tests for `POST /api/v1/evaluate/run`.
core/services/detection_eval.py	New scoring implementation for entity-level metrics.
core/services/data_ingestion.py	Relaxes dataset ingestion from strict whitelist to dataset-shape discovery.
core/repositories/anomaly_repository.py	Adds `run_id` filter to anomaly listing.
core/fastapi_app.py	Registers the new evaluation router.
core/api_routers/evaluation.py	New evaluation endpoint that reads a run’s anomalies and scores them.
core/api_routers/anomalies.py	Adds `run_id` query parameter to anomalies listing endpoint.
.gitignore	Ignores `.venv/`.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    def query_anomalies(self, entity_id=None, model_id=None, run_id=None,
+                        min_risk=None, max_risk=None, limit=1000):
+        '''query anomalies from the platform (optionally scoped to one run)'''


+def query_anomalies(entity_id=None, model_id=None, run_id=None, min_risk=None,
                    max_risk=None, limit=1000):
-    """Query anomalies from the platform."""
+    """Query anomalies from the platform (optionally scoped to one run)."""
    return _get_client().query_anomalies(entity_id=entity_id, model_id=model_id,


+    repo = AnomalyRepository(db)
+    rows = repo.list_all(run_id=body.run_id, limit=100000)
+    anomalies = [
+        {
+            "entity_id": r.entity_id,
+            "risk_score": float(r.risk_score) if r.risk_score is not None else 0.0,
+            "anomaly_type": r.anomaly_type,
+        }
+        for r in rows
+    ]
+    result = evaluate_detections(
+        anomalies=anomalies,
+        malicious_entities=body.malicious_entities,
+        all_entities=body.all_entities,
+        threshold=body.threshold,
+        scenarios=body.scenarios,
+    )
+    logger.info(
+        f"evaluated run {body.run_id}: precision={result.precision} "
+        f"recall={result.recall} over {len(anomalies)} anomalies"
+    )


Jovonni added 5 commits August 22, 2026 16:12

ci(workspace): upgrade pip/setuptools/wheel before sdist installs

7d45969

This branch touches sdk/, which triggers the workspace image build; it fails generating pyspark metadata without modern build tooling. Same fix already validated on the alerts branch.

Jovonni requested a lite review from Copilot August 25, 2026 00:46

Copilot started reviewing on behalf of Jovonni August 25, 2026 00:47 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(data): dataset ingestion + detection evaluation (#35) [WIP] - #146

feat(data): dataset ingestion + detection evaluation (#35) [WIP]#146
Jovonni wants to merge 5 commits into
masterfrom
feat/data-ingestion-testing

Jovonni commented Aug 22, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Jovonni commented Aug 22, 2026

Background

Phase A (this PR so far) — all tested

Tests (18, green locally; run in CI)

Phase B (follow-up in this branch)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants