Skip to content

[python] Use Native REST authorized stream scans - #10579

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-authorized-snapshot-scans
Oct 15, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/native-authorized-snapshot-scans

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Enable the Native backend for the REST authorized streaming reads already supported by PyPaimon. Depends on apache/paimon-rust#1130 reaching Rust main; Native CI continues to build Rust main.

Changes

  • Route standard REST query authorization through Rust's stateful StreamTableScan rather than always using AsyncStreamingTableScan.
  • Keep custom authorization callbacks, custom environments/loaders, and unreproducible IO contexts on the Python path. Validate the actual callback's loader and table identity, including callbacks overridden on a standard environment instance.
  • Include hidden ordinary-predicate operands in Python streaming authorization requests, matching Java's requested column scope.

The authorization, filters, masks, physical row kinds, incremental planning and Blob handling are implemented in Rust core. This PR adds no duplicate Python authorization engine or feature-specific binding API.

Verification

  • 245 Native regression tests passed (4 cases skipped because physical PK retraction tests do not apply to append/Data Evolution tables).
  • 176 existing Python streaming/authorization tests passed.
  • New REST integration cases compare Python and Native for append, primary-key and Data Evolution tables: initial/follow-up reads, changing rules, masks on hidden predicates, row-kind preservation including -U/+U/-D, empty projection, authorization failure/retry, consumer checkpoints, captured policies and combined timestamp-window filters before LIMIT.
  • Regression tests cover instance/subclass callback overrides with a restrictive local policy; the instance override reproduced the routing error before the gate was tightened.
  • Flake8, license headers and Python 3.6 syntax checks passed.

The integration tests use the locally built companion Rust branch. Keep this PR draft until that Rust change has merged so Native CI can exercise it from main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the Python native REST-authorized stream wiring, native capability gating and fallback behavior, predicate/select propagation, row-kind and follow-up frame handling, and custom authorization environments. Targeted tests pass (84 passed, 33 skipped; syntax check passed). No blocking issues found.

@JingsongLi
JingsongLi marked this pull request as ready for review October 15, 2026 07:38
@JingsongLi
JingsongLi merged commit 769164a into apache:master Oct 15, 2026
14 of 16 checks passed
@leaves12138

Copy link
Copy Markdown
Contributor

The Native CI failure is dependency-order related rather than a Python logic regression: this job builds paimon-rust from main, while the REST-authorized streaming/incremental implementation is in apache/paimon-rust#1130. The run therefore reports the Rust client's existing query-auth unsupported error in the new native authorization tests (and related native baseline failures). After #1130 is merged, please rerun this CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants