[python] Enable native nested updates, temporal upserts and descending sequences - #10242
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
Reviewed 32fc718 together with apache/paimon-rust#973 at fab83763a0d20d3adbfee6da71297bbd6ce012bb. No blocking regression found in these paired changes.
Checked native dispatch and fallback boundaries, nested constructor versus safe-assignment semantics, temporal key precision/NULL/duplicate matching, exact source partition scope, and descending sequence ordering against the Java comparator behavior.
Local validation using the Rust extension built from the paired revision and PyArrow 18.1: 973 passed, 2 expected Python-only MAP view-key skips, and 16 passed subtests with all five Native CI flags enabled. An additional 12 constructor and partition boundary cases passed; changed Python files pass flake8. The Rust side also passed 68 focused core/integration tests.
Non-blocking existing limitation found during extra testing: timestamp(ns) partition values with sub-microsecond components can produce missing-file errors through PyPaimon reads. This also reproduces before any upsert on the unchanged base a36c253 with Rust 5c87ec53e8bbaa415047e9a5a2beba27fe1b0acf, so it is not introduced by this PR and should be handled separately.
This is a code-review approval of the paired revisions, not a waiver of the dependency or CI requirements. Keep this PR as a draft until Rust #973 is merged, then rerun Native CI against main and require the remaining checks to pass before merging.
Purpose
Enable more of PyPaimon's supported Parquet read/write/update operations through Native execution, keeping normalization and key matching in Rust core.
Depends on apache/paimon-rust#973. Keep this PR as a draft until the Rust change is merged. Native CI continues to build
apache/paimon-rust@main; it is not redirected to a fork or feature branch.Brief change log
is_nullpredicates. Verify that both paths match exact composite input partitions and do not read unrelated data files.The existing Python wrappers and Native APIs are reused; no new public binding methods are introduced.
Tests
Built the extension from the dependent Rust branch and ran these suites with PyArrow 18.1 and all Native CI flags enabled:
Result: 973 passed, 2 skipped, 16 subtests passed. The two skips are Python-only MAP view-key cases where PyArrow 18.1 lacks an Arrow
takekernel; their Native counterparts pass.Native update coverage counters: row-ID 167, grouped 54, predicate 77, upsert 46, incremental 15. Changed Python files pass flake8. The dependent Rust branch passes 68 focused core tests, workspace Clippy and rustfmt.