Add nullable UnionArray

[connortsui20]

Tracking issue: #8769

Implements the array portion of #8769 on top of #8791.

• adds the canonical sparse UnionArray
• stores outer union validity exclusively in the nullable u8 type IDs child
• treats child validity as inner/selected-value nullability, not outer validity
• ignores the type-ID payload at outer-null rows during validation
• preserves outer and inner nulls through scalar_at, serde, constants, chunking, filtering, slicing, masking, recursive canonicalization, and compression
• recursively builds valid defaults for unselected children and returns an error for genuinely uninhabited schemas
• canonicalizes empty unions without inventing placeholder values
• supports non-nullable-index take; nullable-index take remains an explicit unsupported error for now

[codspeed-hq]

Merging this PR will improve performance by 11.17%

⚡ 1 improved benchmark
✅ 1669 untouched benchmarks
⏩ 52 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	true_count_vortex_buffer[128]	580.6 ns	522.2 ns	+11.17%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing agent/union-array-nullability (16dddab) with agent/union-scalar-nullability (d5458d2)²}

Footnotes

1. 52 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on agent/union-scalar-nullability (abefbe3) during the generation of this report, so 9d3b3cc was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[github-actions]

Polar Signals Profiling Results

Latest Run

Status	Commit	Job	Attempt	Link
🟢 Done	16dddab		1	Explore Profiling Data

---

Powered by Polar Signals Cloud

[github-actions]

Benchmarks: Vortex queries 📖

Verdict: No clear signal (low confidence)
Attributed Vortex impact: +3.1%
Engines: DataFusion No clear signal (+6.1%, low confidence) · DuckDB No clear signal (+0.2%, low confidence)
Vortex (geomean): 1.007x ➖
Parquet (geomean): 0.992x ➖
Shifts: Parquet (control) -0.8% · Median polish +0.4%

How to read Verdict and Engines

• Verdict: Overall PR-level signal after subtracting baseline drift estimated from Parquet control rows. It can be Likely improvement, Likely regression, or No clear signal.
• Engines: Per-engine attribution. DataFusion is compared against DataFusion/Parquet controls; DuckDB is compared against DuckDB/Parquet controls. This answers whether each engine improved or regressed independently.
• Confidence: Based on directional consistency, share of rows above the noise floor, and control-run noise.

---

datafusion / vortex-file-compressed (1.040x ➖, 0↑ 0↓)

name	PR 16dddab (ns)	base 7cf53cb (ns)	ratio (PR/base)
vortex_q00/datafusion:vortex-file-compressed	9911242	9632925	1.03
vortex_q01/datafusion:vortex-file-compressed	6359580	6047747	1.05

datafusion / parquet (0.980x ➖, 0↑ 0↓)

name	PR 16dddab (ns)	base 7cf53cb (ns)	ratio (PR/base)
vortex_q00/datafusion:parquet	20198573	20532528	0.98
vortex_q01/datafusion:parquet	4715465	4826794	0.98

duckdb / vortex-file-compressed (1.005x ➖, 0↑ 0↓)

name	PR 16dddab (ns)	base 7cf53cb (ns)	ratio (PR/base)
vortex_q00/duckdb:vortex-file-compressed	10209639	10341086	0.99
vortex_q01/duckdb:vortex-file-compressed	6361293	6212619	1.02

duckdb / parquet (1.003x ➖, 0↑ 0↓)

name	PR 16dddab (ns)	base 7cf53cb (ns)	ratio (PR/base)
vortex_q00/duckdb:parquet	23487352	23424912	1.00
vortex_q01/duckdb:parquet	9491128	9452056	1.00

---

No file size changes detected.