Skip to content

Support REST authorized streaming and incremental reads - #1130

Merged
JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-authorized-snapshot-scans
Oct 15, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
JingsongLi:codex/native-authorized-snapshot-scans

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

PyPaimon already supports REST query authorization for streaming and timestamp-window reads, but Native StreamTableScan and combined incremental planning currently reject authorized tables. Authorized row-kind reads also reject every row filter or mask.

Changes

  • Authorize every stream plan() call, including empty polls, before restoring consumers or advancing a snapshot. Keep the grant on an ephemeral scan and the resulting splits; refresh it on the next call.
  • Reuse that scope for combined Delta/Changelog and timestamp-window planning. Validate planned files before attaching the grant.
  • Preserve physical change kinds through row filters and column masks, including hidden fields and empty projections. Reuse the existing authorized Blob pipeline and apply LIMIT after filtering.
  • Keep the raw per-snapshot incremental and Diff authorization restrictions.

The lifecycle follows Java AbstractDataTableScan.plan() / applyAuthRules() / convertPlan(). No new public API or dependency is introduced.

Verification

  • 350 focused Rust tests passed across REST catalog, streaming, combined incremental/timestamp scans, partition filters, scan units and read units.
  • New tests first reproduced the existing authorization rejection, then cover changing rules, captured rules, initial/follow-up/empty-poll failures and retry, consumer progress, schema/UUID drift, physical -U/+U/-D, zero projections, and Blob descriptors excluded before resolution.
  • Built the Python extension and exercised it with PyPaimon's Native and authorized-read regression suites.
  • cargo fmt --all --check
  • cargo +1.98.0 clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings

Companion PyPaimon PR: apache/paimon#10579. It enables standard REST authorized Native stream scans after this change reaches Rust main.

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed REST query authorization across streaming and incremental reads, including per-poll grant refresh, split-scoped grants, retry/checkpoint behavior on authorization failures, row-kind reads, masks/filters, limits, schema or identity drift, and blob resolution. Targeted tests pass (7 Rust authorization tests; CI is still running). No blocking issues found.

@JingsongLi
JingsongLi merged commit f0f5d96 into apache:main Oct 15, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants