Skip to content

feat: Optimize PG graph database wipes - BED-8787#2936

Open
StephenHinck wants to merge 3 commits into
mainfrom
BED-8787
Open

feat: Optimize PG graph database wipes - BED-8787#2936
StephenHinck wants to merge 3 commits into
mainfrom
BED-8787

Conversation

@StephenHinck

@StephenHinck StephenHinck commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Description

Implements the new WipeGraph function supported optionally on DAWGS drivers to execute rapid graph wiping if supported. Initially, this is supported on PostgreSQL drivers utilizing TRUNCATE on node/edge tables.

This PR requires the merging/release of DAWGS from SpecterOps/DAWGS#99 to realize the optimization gains, however merging may occur in any order as the implementation is optional on all graph drivers.

Motivation and Context

Resolves BED-8787

Why is this change required? What problem does it solve?

How Has This Been Tested?

Validated in local environments, and additional integration tests added to DAWGS.

Screenshots (optional):

Utilizing a test dataset from OGGen with 1M nodes, 3 tiers, and 10 edges between each tier showed a 99% reduction in deletion time.

Pre-change:

time=2026-06-30T15:23:52.946Z level=INFO message=DeleteCollectedGraphData delete_all_data=true delete_sourceless_data=false delete_source_kinds="" delete_relationships="" measurement_id=75 elapsed=33.178734682s

Post-change:

time=2026-06-30T01:02:54.377Z level=INFO message=DeleteCollectedGraphData delete_all_data=true delete_sourceless_data=false delete_source_kinds="" delete_relationships="" measurement_id=156 elapsed=115.218083ms

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist:

Summary by CodeRabbit

  • Bug Fixes
    • Improved full-graph deletion to preserve the database’s migration version, keeping required system state consistent after a complete wipe.
    • Enabled supported graph backends to clear all graph data via an optimized wipe path while keeping migration metadata intact.
  • Tests
    • Added/extended an integration test to verify the migration version is unchanged before and after a full graph wipe.

@StephenHinck StephenHinck self-assigned this Jun 29, 2026
@coderabbitai

coderabbitai Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

delete.go adds a graphWiper fast path for full graph deletion that preserves MigrationData by reading it before the wipe and recreating it inside the wipe transaction. The integration test now checks that the version stays the same after deletion.

Changes

Bulk graph wipe preserving MigrationData

Layer / File(s) Summary
graphWiper interface, dispatch, and wipeAllGraphData implementation
cmd/api/src/daemons/datapipe/delete.go
Adds the migrations import, defines graphWiper with WipeGraph(ctx, retain), routes DeleteAllGraph through the new fast path when supported, and implements wipeAllGraphData to read and recreate MigrationData during the wipe.
Integration test: MigrationData version preserved after wipe
cmd/api/src/daemons/datapipe/delete_integration_test.go
Imports migrations and extends TestDeleteAllData to record the pre-wipe MigrationData version and assert it matches the post-wipe version.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: optimizing PostgreSQL graph database wipes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description check ✅ Passed The PR description follows the required template and includes the key sections, ticket reference, testing notes, and change type.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch BED-8787

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cmd/api/src/daemons/datapipe/delete.go`:
- Around line 183-194: The truncate path in delete.go is treating
migrations.ErrNoMigrationData from GetMigrationData as safe to continue, but
that helper also uses the same sentinel for read/property failures and returns a
fallback version; update the logic around GetMigrationData in the wipe flow to
only proceed when a truly missing MigrationData node is confirmed, and abort on
any ambiguous or malformed read so you do not recreate MigrationData from the
lossy fallback. Use the existing GetMigrationData and ErrNoMigrationData
handling to split “missing node” from “failed read” before the graph wipe.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 7fb0c2f4-4cd2-488b-a96e-a80bfca00d30

📥 Commits

Reviewing files that changed from the base of the PR and between 3a3f093 and 0e89fa9.

📒 Files selected for processing (2)
  • cmd/api/src/daemons/datapipe/delete.go
  • cmd/api/src/daemons/datapipe/delete_integration_test.go

Comment thread cmd/api/src/daemons/datapipe/delete.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant