Skip to content

feat: add deletion vector compaction planner - #918

Draft
wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-planner
Draft

feat: add deletion vector compaction planner#918
wirybeaver wants to merge 3 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-planner

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026

Copy link
Copy Markdown

Summary

  • add a snapshot-bound, metadata-only CompactionPlanner without rewrite execution
  • select small files and files with qualifying, deduplicated file-scoped position-delete pressure; equality and partition-scoped deletes are excluded
  • reject duplicate data-file scan tasks before candidate counting and require qualifying delete pressure for oversized files
  • validate configuration, non-negative source snapshot IDs, and file metadata; reject missing partition spec IDs and invalid DV cardinality; cap position-delete cardinality and return checked-arithmetic errors
  • group null- and NaN-containing partitions with canonical Iceberg equality semantics and emit deterministic partition/file order
  • compare near-INT64_MAX size and delete-ratio thresholds without lossy integer-to-double conversion
  • use deterministic best-fit-decreasing packing to shape partition groups by target_file_size_bytes

API for Ticket 08

  • planning requires CompactionPlanner::Plan(int64_t source_snapshot_id, scan_tasks, config)
  • negative source snapshot IDs are rejected
  • CompactionPlan::source_snapshot_id records the snapshot used to produce the scan tasks
  • an executor must validate that source snapshot before rewriting, so a stale plan cannot silently resurrect rows deleted after planning

This PR is stacked on #913 and #915. Until those dependencies merge, it contains their prerequisite commits; review commit 56ed503 for Ticket 07 itself. This replaces Ticket 07 commit 03990086 (which had already replaced f05db6b), and Ticket 08 must synchronize to 56ed50369dcdfedd7af5962493ecfc33af0420af.

Part of #916.

Test Plan

  • GCC 14 build: cmake --build build --target compaction_planner_test util_test -j2
  • GCC 14 tests: ctest --test-dir build -R "^(compaction_planner_test|util_test)$" --output-on-failure (2/2 passed)
  • planner tests (18 total) cover config and threshold boundaries, zero-threshold semantics, snapshot binding and negative IDs, duplicate data-file tasks, null and canonical NaN partitions, missing spec IDs, deterministic ordering, mixed delete kinds, delete deduplication/DV ranges, cardinality capping and validation, checked overflow, exact near-INT64_MAX comparisons, oversized files, partition isolation, small files, target-sized grouping, and best-fit 60,60,40,40 packing
  • /home/user/.cache/uv/archive-v0/1n0bbgciCQ9iE9BH/bin/pre-commit run --all-files (all hooks passed)
  • git diff --check (passed)

Revert Plan

Revert commit 56ed50369dcdfedd7af5962493ecfc33af0420af and restore the prior Ticket 07 dependency in downstream branches.

Add a snapshot-bound metadata-only planner that selects rewrite candidates by file size and deduplicated file-scoped position-delete pressure. Validate planner inputs and group candidates deterministically within partitions using target-sized groups.
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-planner branch from 0399008 to 56ed503 Compare September 5, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant