Search before asking
Motivation
PyPaimon reconstructs MAP values when reading ordinary MAP columns and shared-shredding MAP columns. These paths convert Arrow arrays to Python objects and process mapping entries or keys row by row. For large batches, this introduces CPU overhead even after unrelated columns have been pruned.
The same module contains three reconstruction paths with opportunities to reduce this overhead. This umbrella issue tracks them as separate changes so that each can be reviewed and benchmarked independently.
Solution
Preserve null handling, missing-key behavior, sliced-array offsets, and the existing selection and ordering semantics. Keep fallback paths where Arrow operations are unavailable or do not improve performance.
Anything else?
Submit a separate PR for each item, with correctness coverage and before/after benchmarks. These changes concern read-side reconstruction and require no file-format or public API change.
Are you willing to submit a PR?
Search before asking
Motivation
PyPaimon reconstructs MAP values when reading ordinary MAP columns and shared-shredding MAP columns. These paths convert Arrow arrays to Python objects and process mapping entries or keys row by row. For large batches, this introduces CPU overhead even after unrelated columns have been pruned.
The same module contains three reconstruction paths with opportunities to reduce this overhead. This umbrella issue tracks them as separate changes so that each can be reviewed and benchmarked independently.
Solution
assemble_shared_shredding_selected_keys: use Arrow operations to resolve requested keys and select values, with scalar fallback for small batches and expensive candidate mappings. PR: [python] Vectorize selected-key reconstruction for shared-shredding MAPs #10143.assemble_normal_map_selected_keys: use Arrow key lookup to avoid converting and searching each row's full MAP in Python where supported.assemble_shared_shredding_map: batch the integer field-mapping conversion while preserving MAP entry order and value handling.Preserve null handling, missing-key behavior, sliced-array offsets, and the existing selection and ordering semantics. Keep fallback paths where Arrow operations are unavailable or do not improve performance.
Anything else?
Submit a separate PR for each item, with correctness coverage and before/after benchmarks. These changes concern read-side reconstruction and require no file-format or public API change.
Are you willing to submit a PR?