Skip to content

[python] Enable native nested updates, temporal upserts and descending sequences - #10242

Merged
JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-nested-updates
Sep 27, 2026
Merged

JingsongLi merged 2 commits into
apache:masterfrom
JingsongLi:codex/native-nested-updates

Conversation

@JingsongLi

@JingsongLi JingsongLi commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Purpose

Enable more of PyPaimon's supported Parquet read/write/update operations through Native execution, keeping normalization and key matching in Rust core.

Depends on apache/paimon-rust#973. Keep this PR as a draft until the Rust change is merged. Native CI continues to build apache/paimon-rust@main; it is not redirected to a fork or feature branch.

Brief change log

  • Enable ARRAY/MAP/ROW columns for direct/grouped row-ID updates, incremental row-ID writers, predicate assignments and key upserts after completing the required core normalization and conversion.
  • Enable TIME, TIMESTAMP and TIMESTAMP_LTZ upsert keys, including Arrow and row input, NULL keys, duplicate source/target keys and nanosecond precision.
  • Enable descending non-floating user sequence fields through the existing Rust ordering implementation. Floating sequence fields retain their existing fallback.
  • Fix NULL partition upserts on the Python path by constructing is_null predicates. Verify that both paths match exact composite input partitions and do not read unrelated data files.
  • Add batch/stream and Python/Native interoperability coverage across deduplicate, partial-update and aggregation sequence ordering, including compound sequence fields, NULL ordering, hidden sequence projections and post-merge predicates.
  • Assert that enabled cases cannot silently fall back to Python, and verify committed results through both Python and Native readers.

The existing Python wrappers and Native APIs are reused; no new public binding methods are introduced.

Tests

Built the extension from the dependent Rust branch and ran these suites with PyArrow 18.1 and all Native CI flags enabled:

pypaimon/tests/native_nested_update_test.py
pypaimon/tests/native_upsert_capabilities_test.py
pypaimon/tests/native_sequence_test.py
pypaimon/tests/native_update_test.py
pypaimon/tests/table_update_test.py
pypaimon/tests/table_update_by_row_id_chunked_test.py
pypaimon/tests/native_write_capabilities_test.py
pypaimon/tests/native_write_test.py

Result: 973 passed, 2 skipped, 16 subtests passed. The two skips are Python-only MAP view-key cases where PyArrow 18.1 lacks an Arrow take kernel; their Native counterparts pass.

Native update coverage counters: row-ID 167, grouped 54, predicate 77, upsert 46, incremental 15. Changed Python files pass flake8. The dependent Rust branch passes 68 focused core tests, workspace Clippy and rustfmt.

@JingsongLi JingsongLi changed the title [python] Enable native updates and upserts for nested columns [python] Enable native nested updates, temporal upserts and descending sequences Sep 27, 2026

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 32fc718 together with apache/paimon-rust#973 at fab83763a0d20d3adbfee6da71297bbd6ce012bb. No blocking regression found in these paired changes.

Checked native dispatch and fallback boundaries, nested constructor versus safe-assignment semantics, temporal key precision/NULL/duplicate matching, exact source partition scope, and descending sequence ordering against the Java comparator behavior.

Local validation using the Rust extension built from the paired revision and PyArrow 18.1: 973 passed, 2 expected Python-only MAP view-key skips, and 16 passed subtests with all five Native CI flags enabled. An additional 12 constructor and partition boundary cases passed; changed Python files pass flake8. The Rust side also passed 68 focused core/integration tests.

Non-blocking existing limitation found during extra testing: timestamp(ns) partition values with sub-microsecond components can produce missing-file errors through PyPaimon reads. This also reproduces before any upsert on the unchanged base a36c253 with Rust 5c87ec53e8bbaa415047e9a5a2beba27fe1b0acf, so it is not introduced by this PR and should be handled separately.

This is a code-review approval of the paired revisions, not a waiver of the dependency or CI requirements. Keep this PR as a draft until Rust #973 is merged, then rerun Native CI against main and require the remaining checks to pass before merging.

@JingsongLi
JingsongLi marked this pull request as ready for review September 27, 2026 14:57
@JingsongLi
JingsongLi merged commit 1651043 into apache:master Sep 27, 2026
12 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants