[SPARK-58023][PYTHON] Fast paths for converting Arrow string, binary, numeric and boolean columns to Python rows

[viirya]

What changes were proposed in this pull request?

Follow-up of SPARK-58019 (#57099); only the last commit is new — this PR is stacked on #57099 and will be rebased once it merges.

Extend ArrowTableToRowsConversion._to_pylist with fast paths for leaf (non-nested) columns:

• string/large_string/binary/large_binary columns use Arrow's object-dtype NumPy conversion, which produces exactly str/bytes/None — no other type can come out of it.
• integer/float32/float64/boolean columns without nulls convert via a zero-copy NumPy view (booleans are bit-packed, hence copied) and ndarray.tolist(), which materializes exact Python int/float/bool. With nulls, values are filled with a placeholder first (pc.fill_null) and nulls are restored to None from the validity bitmap, so ints are always materialized from the original int buffer, never through a float representation.

Types whose as_py returns non-primitive objects (dates, timestamps, decimals, ...) keep using to_pylist. Since list columns convert their flattened child values through _to_pylist, list-typed columns get the leaf speedup on top of the SPARK-58019 bulk offsets slicing.

Why are the changes needed?

After SPARK-58019, leaf values are still converted one Scalar at a time by PyArrow's to_pylist() (apache/arrow#50326). ASV microbenchmark (bench_arrow.ArrowLeafColumnToRowsBenchmark, 1M rows, PyArrow 24.0.0):

case	to_pylist()	this PR	speedup
string, 10% nulls	196 ms	20 ms	9.7x
int64, 10% nulls	99 ms	28 ms	3.6x
float64, no nulls	100 ms	9 ms	11x

End-to-end list<string> conversion (ArrowListColumnToRowsBenchmark, 1M rows) improves from 507 ms (SPARK-58019) to 118 ms.

Does this PR introduce any user-facing change?

No. Only performance; conversion results are identical (covered by exact-type tests).

How was this patch tested?

Extended ArrowColumnToPylistTests with flat string/large_string/binary/large_binary, int8/int64/uint64 (including values beyond 2^62), float32/float64 (NaN/inf), bool, no-null and all-null variants, sliced and chunked views — all compared against column.to_pylist() with exact element-type assertions — plus non-primitive leaves (date/timestamp/decimal) exercising the fallback, and lists of fast-path leaves. New ASV benchmark class ArrowLeafColumnToRowsBenchmark.

Was this patch authored or co-authored using generative AI tooling?

Yes. This pull request and its description were written by Isaac (Claude Code).

[viirya]

Closing this PR. Rationale:

During review of the sibling PRs, concerns were raised about the maintainability of routing data values through NumPy (behavior across the supported arrow/numpy version matrix, and keeping the "no type coercion" invariant guarded by tests over time). While the conversions here are exactness-tested — the object-dtype conversion for strings/binary can only produce str/bytes/None, and nullable numerics are filled and restored from the validity bitmap rather than going through a float representation — this is also precisely the layer that apache/arrow#50327 implements properly in C, planned for PyArrow 25.0.1 per the Arrow dev-list discussion.

With the version gate added in SPARK-58019 (#57099), Spark automatically uses PyArrow's native conversion once the installed PyArrow contains that fix — so the leaf-level speedup reaches users through a PyArrow upgrade rather than through NumPy-based code maintained here. The remaining Spark-side PRs are scoped to what Arrow cannot do for Spark: SPARK-58019/SPARK-58024 (bulk offsets/validity slicing; no data values through NumPy) and SPARK-58050 (bulk result assembly; no NumPy at all).

For the record: this PR gave flat string 196ms→20ms and int64 99ms→28ms per 1M rows, and its removal lowers the struct/map gains in SPARK-58024 from 5.2x/6.8x to 2.0x/2.7x until the PyArrow fix ships. We accept that interim gap in exchange for the smaller maintenance surface.