Skip to content

Support explicit partition selectors in native table scans - #1129

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-partition-selectors
Oct 15, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-partition-selectors

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Allow PyPaimon's existing explicit partition selectors to use native planning, including REST tables with query authorization. Previously the binding only exposed ordinary row filters, so PyPaimon combined the two filters or fell back to Python for authorized scans.

Match Java ReadBuilderImpl.configureScan and AbstractDataTableScan.rejectMaskedPartitionFilter: an explicit partition selector is scan-only, replaces derived partition pruning, and must reject masked operands. Ordinary predicates remain read filters.

PyPaimon integration: apache/paimon#10576 (continues building Rust main).

Brief change log

  • Add core ReadBuilder.with_partition_filter(Option<Predicate>) with table-field validation, replacement/clearing semantics, and typed Python binding/stub. Batch, incremental and stream scans capture the same selector.
  • Preserve ordinary partition conditions when an explicit selector overrides derived pruning. Evaluate those conditions against split partition metadata, so files may omit physical partition columns and independent readers still enforce their filters.
  • Keep split authorization validation before read-side partition selection. Preserve the ordinary condition in row-kind, incremental/audit and Diff worker paths.
  • Test append, primary-key and data-evolution tables, reordered partition fields, hidden predicates, LIMIT, invalid inputs, checkpoint/restore, Diff images, REST authorization scopes, and masked selectors.

Tests

  • ReadBuilder unit tests: 49 passed.
  • Partition selector, batch/changelog/Diff incremental, audit-log and stream integration suites: 92 passed.
  • REST query authorization tests: 38 passed.
  • Python binding read and stream suites against a freshly rebuilt extension: 163 passed; final partition-specific rerun: 7 passed.
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings: passed.
  • cargo fmt --all --check and git diff --check: passed.

API and Format

Adds a core and Python ReadBuilder setter. Explicit selectors use table-field indices and only accept partition fields; None clears the selector. Format tables remain outside this API. No storage/wire format changes.

Documentation

Core API rustdoc and Python type stubs describe scan-only selection, ordinary filtering and authorization behavior.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 4270044 into apache:main Oct 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants