[SPARK-58024][PYTHON] Convert Arrow struct and map columns to Python rows in bulk by viirya · Pull Request #57105 · apache/spark

viirya · 2026-07-08T00:46:02Z

What changes were proposed in this pull request?

Follow-up of SPARK-58019 (#57099); only the last commit is new — this PR is stacked on #57099 and will be rebased once it merges. (Originally stacked on SPARK-58023 (#57104), which has been closed in favor of the upstream apache/arrow#50326 fix; the numbers below are measured without it.)

Extend ArrowTableToRowsConversion._to_pylist with bulk paths for struct and map columns:

Struct columns convert each child field in bulk (recursively reusing the bulk list paths), then zip the field values into one dict per row, masked by the validity bitmap. Duplicate field names fall back to to_pylist, so they raise the same ValueError that StructScalar.as_py raises.
Map columns share the list offsets layout: the flattened keys and items children are each converted in bulk, and every row becomes a list of (key, value) tuples, matching MapScalar.as_py exactly.

Why are the changes needed?

Struct and map columns still convert one Scalar per row; per-row Scalar/wrapper allocation dominates (apache/arrow#50326), and maps are the worst case since every row also wraps its keys/items in per-row Arrays. ASV microbenchmark (bench_arrow.ArrowStructMapColumnToRowsBenchmark, 1M rows, 10% nulls, PyArrow 24.0.0):

case	`to_pylist()`	this PR	speedup
`struct<int64,string>`	970 ms	474 ms	2.0x
`map<string,int64>` (2 entries/row)	2.16 s	799 ms	2.7x

Peak memory unchanged.

Does this PR introduce any user-facing change?

No. Only performance; conversion results are identical (covered by exact-type tests).

How was this patch tested?

Extended ArrowColumnToPylistTests with struct (incl. nested struct/list children, empty struct, all-null), map (incl. list values), struct-of-map, list-of-struct, sliced and chunked views — all compared against column.to_pylist() with exact element-type assertions — plus dedicated tests that duplicate struct field names still raise ValueError and that empty-struct rows are distinct dict objects. New ASV benchmark class ArrowStructMapColumnToRowsBenchmark.

Was this patch authored or co-authored using generative AI tooling?

Yes. This pull request and its description were written by Isaac (Claude Code).

…rows in bulk Extend ArrowTableToRowsConversion._to_pylist with bulk paths for struct and map columns: * Struct columns convert each child field in bulk (recursively reusing the list/leaf fast paths), then zip the field values into one dict per row, masked by the validity bitmap. Duplicate field names fall back to to_pylist so they raise the same ValueError that StructScalar.as_py raises. * Map columns share the list offsets layout: the flattened keys and items children are each converted in bulk and every row becomes a list of (key, value) tuples, matching MapScalar.as_py exactly. ASV microbenchmark (bench_arrow.ArrowStructMapColumnToRowsBenchmark, 1M rows, 10% nulls): struct<int64,string> 914ms -> 175ms (5.2x); map<string,int64> with 2 entries per row 2.07s -> 303ms (6.8x). Peak memory unchanged. Co-authored-by: Isaac

viirya · 2026-07-10T23:27:51Z

@huaxingao Can you hold 4.2 new RC cut until this gets merged? This is to extend #57099 to struct and map columns. Thanks.

viirya · 2026-07-10T23:29:20Z

@dongjoon-hyun @Yicong-Huang @gaogaotiantian This extends the optimization to struct and map columns. Could you also take a look? Thanks!

gaogaotiantian · 2026-07-11T00:32:49Z

+                for i in range(n)
+            ]
+
+        if pa.types.is_list(column.type) or pa.types.is_large_list(column.type):


nit: could you change this if and the one below to elif? otherwise looks good to me.

Done in fc45dd7 — both changed to elif.

Co-authored-by: Isaac

…rows in bulk ### What changes were proposed in this pull request? Follow-up of SPARK-58019 (#57099); **only the last commit is new — this PR is stacked on #57099 and will be rebased once it merges.** (Originally stacked on SPARK-58023 (#57104), which has been closed in favor of the upstream apache/arrow#50326 fix; the numbers below are measured without it.) Extend `ArrowTableToRowsConversion._to_pylist` with bulk paths for struct and map columns: - **Struct** columns convert each child field in bulk (recursively reusing the bulk list paths), then zip the field values into one dict per row, masked by the validity bitmap. Duplicate field names fall back to `to_pylist`, so they raise the same `ValueError` that `StructScalar.as_py` raises. - **Map** columns share the list offsets layout: the flattened keys and items children are each converted in bulk, and every row becomes a list of `(key, value)` tuples, matching `MapScalar.as_py` exactly. ### Why are the changes needed? Struct and map columns still convert one Scalar per row; per-row Scalar/wrapper allocation dominates (apache/arrow#50326), and maps are the worst case since every row also wraps its keys/items in per-row Arrays. ASV microbenchmark (`bench_arrow.ArrowStructMapColumnToRowsBenchmark`, 1M rows, 10% nulls, PyArrow 24.0.0): | case | `to_pylist()` | this PR | speedup | |---|---|---|---| | `struct<int64,string>` | 970 ms | 474 ms | 2.0x | | `map<string,int64>` (2 entries/row) | 2.16 s | 799 ms | 2.7x | Peak memory unchanged. ### Does this PR introduce _any_ user-facing change? No. Only performance; conversion results are identical (covered by exact-type tests). ### How was this patch tested? Extended `ArrowColumnToPylistTests` with struct (incl. nested struct/list children, empty struct, all-null), map (incl. list values), struct-of-map, list-of-struct, sliced and chunked views — all compared against `column.to_pylist()` with exact element-type assertions — plus dedicated tests that duplicate struct field names still raise `ValueError` and that empty-struct rows are distinct dict objects. New ASV benchmark class `ArrowStructMapColumnToRowsBenchmark`. ### Was this patch authored or co-authored using generative AI tooling? Yes. This pull request and its description were written by Isaac (Claude Code). Closes #57105 from viirya/arrow-to-pylist-maps-structs. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 9c6e57b) Signed-off-by: Liang-Chi Hsieh <viirya@gmail.com>

viirya · 2026-07-11T05:40:31Z

Merge Summary:

merged into master 9c6e57b
merged into branch-4.x 5974091
merged into branch-4.2 44daf01

Posted by merge_spark_pr.py

viirya · 2026-07-11T05:40:56Z

Thanks @gaogaotiantian

viirya · 2026-07-11T05:44:57Z

@huaxingao Both PRs are now merged into master: #57099 (SPARK-58019) and this #57105 (SPARK-58024). Nothing else is blocking on our side, so please feel free to proceed with the new 4.2 RC cut. Thank you for your patience and for holding the RC for these!

viirya force-pushed the arrow-to-pylist-maps-structs branch 3 times, most recently from 8b51d7a to 1cf3d8a Compare July 8, 2026 06:34

viirya mentioned this pull request Jul 8, 2026

[SPARK-58050][PYTHON] Bulk-assemble Arrow Python UDF results #57137

Closed

viirya force-pushed the arrow-to-pylist-maps-structs branch from 1cf3d8a to f3340f4 Compare July 8, 2026 20:09

Yicong-Huang mentioned this pull request Jul 9, 2026

[SPARK-58016][PYTHON] Revert Arrow-optimized Python UDFs and Arrow-based PySpark IPC by default #57094

Closed

viirya force-pushed the arrow-to-pylist-maps-structs branch 5 times, most recently from 904fa86 to 96a606f Compare July 10, 2026 21:22

viirya force-pushed the arrow-to-pylist-maps-structs branch from 96a606f to 40f92f7 Compare July 10, 2026 23:26

viirya requested review from Yicong-Huang and gaogaotiantian and removed request for Yicong-Huang July 10, 2026 23:27

gaogaotiantian reviewed Jul 11, 2026

View reviewed changes

[SPARK-58024][PYTHON] Address review: use elif for type branches

fc45dd7

Co-authored-by: Isaac

gaogaotiantian approved these changes Jul 11, 2026

View reviewed changes

viirya closed this in 9c6e57b Jul 11, 2026

viirya deleted the arrow-to-pylist-maps-structs branch July 11, 2026 05:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[SPARK-58024][PYTHON] Convert Arrow struct and map columns to Python rows in bulk#57105

[SPARK-58024][PYTHON] Convert Arrow struct and map columns to Python rows in bulk#57105
viirya wants to merge 2 commits into
apache:masterfrom
viirya:arrow-to-pylist-maps-structs

viirya commented Jul 8, 2026 •

edited

Loading

Uh oh!

viirya commented Jul 10, 2026

Uh oh!

viirya commented Jul 10, 2026

Uh oh!

gaogaotiantian Jul 11, 2026 •

edited

Loading

Uh oh!

viirya Jul 11, 2026

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

viirya commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

viirya commented Jul 10, 2026

Uh oh!

viirya commented Jul 10, 2026

Uh oh!

gaogaotiantian Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

viirya Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

viirya commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

viirya commented Jul 8, 2026 •

edited

Loading

gaogaotiantian Jul 11, 2026 •

edited

Loading