Skip to content

feat: add deletion vector compaction execution - #921

Draft
wirybeaver wants to merge 6 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor
Draft

feat: add deletion vector compaction execution#921
wirybeaver wants to merge 6 commits into
apache:mainfrom
wirybeaver:feature/dv-compaction-executor

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026

Copy link
Copy Markdown

Summary

  • add a narrow CompactionExecutor API that consumes snapshot-bound planner groups and rewrites live rows through FileScanTaskReader and DataWriter
  • refresh before writing, reject stale plans, and use ValidateFromSnapshot(source_snapshot_id) to prevent concurrent deletes from being resurrected
  • atomically replace all planned data files while removing superseded file-scoped Parquet position-delete and deletion-vector entries
  • preserve data sequence numbers and v3 row lineage, including files written before a v2-to-v3 table upgrade
  • validate source-file uniqueness before writing and retain shared Puffin objects
  • define FileIO::DeleteFile as idempotent for missing paths and enforce the contract in Arrow and test implementations, allowing the executor to own output paths before creation and safely clean up failures at any writer-construction stage while retaining real deletion failures
  • rebuild the stack on final Ticket 04 commit 477a2ff, replay final Ticket 07 source 56ed503 exactly as 5e0c98b, and keep executor work in one commit

Closes #916.

Depends on #913, #914, #915, #918, and #919.

Test plan

  • cmake --build build --target data_test compaction_planner_test arrow_test -j3 — passed
  • ./build/src/iceberg/test/data_test --gtest_filter='CompactionExecutor*' — 12 passed
  • ./build/src/iceberg/test/data_test — 192 passed
  • ./build/src/iceberg/test/compaction_planner_test — 18 passed
  • ./build/src/iceberg/test/arrow_test — 83 passed
  • pre-commit run -a — all hooks passed

Executor coverage includes physical row removal, no dangling delete-manifest entries, shared Puffin safety, duplicate-plan rejection, stale-plan rejection, multi-file/multi-group atomicity, cleanup failure and retry, writer failure before creation, OutputFile::Create failure before materialization, failure after file creation, executor reuse, commit-state-unknown ownership, file-scoped Parquet delete removal, and upgraded-table row lineage. FileIO coverage verifies idempotent deletion of missing files for Arrow and the standard test implementation.

@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch 3 times, most recently from 68f482c to ab5625a Compare September 5, 2026 23:32
Add a snapshot-bound metadata-only planner that selects rewrite candidates by file size and deduplicated file-scoped position-delete pressure. Validate planner inputs and group candidates deterministically within partitions using target-sized groups.
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from ab5625a to 55c6b56 Compare September 6, 2026 00:04
@wirybeaver
wirybeaver force-pushed the feature/dv-compaction-executor branch from 55c6b56 to b3834d9 Compare September 6, 2026 00:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Complete deletion vector feature support

1 participant