Skip to content

fix: support all non-negative int64 deletion vector positions - #917

Draft
wirybeaver wants to merge 2 commits into
apache:mainfrom
wirybeaver:fix/dv-full-positive-int64
Draft

fix: support all non-negative int64 deletion vector positions#917
wirybeaver wants to merge 2 commits into
apache:mainfrom
wirybeaver:fix/dv-full-positive-int64

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026

Copy link
Copy Markdown

Summary

  • replace dense high-32-bit bitmap storage with a sparse ordered map
  • support every non-negative int64_t position through INT64_MAX without allocating intervening buckets
  • serialize only populated buckets in ascending key order and normalize empty decoded buckets
  • document that the half-open range API cannot include INT64_MAX, which must be added separately
  • avoid overflow when a coalesced position-delete range ends at INT64_MAX
  • reject deletion-vector payloads that exceed the signed 32-bit framing limit before allocating the complete serialized payload
  • preflight every merged position index before creating the Puffin output, so an oversized first or later vector cannot leave an empty or partial file

Tracking issue: #916

Depends on #913. This branch includes the cross-language fixture commit from that PR and should remain draft until the dependency is resolved.

Test plan

  • build/src/iceberg/test/util_test --gtest_filter='RoaringPositionBitmapTest.*:RoaringPositionBitmapInteropTest.*:PositionDeleteIndexTest.*:PositionDeleteRangeConsumerTest.*:DVWriterPreflightTest.*' (65 passed)
  • build/src/iceberg/test/puffin_test (31 passed, including 5 cross-language DV fixtures)
  • pre-commit run --all-files (passed)

@wirybeaver
wirybeaver force-pushed the fix/dv-full-positive-int64 branch from 0a6cc1a to 5c494b6 Compare September 5, 2026 23:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant