Skip to content

fix: preserve constructor field metadata through coercion - #27502

Open
Xuanwo wants to merge 14 commits into
apache:mainfrom
Xuanwo:xuanwo/constructor-field-metadata
Open

Xuanwo wants to merge 14 commits into
apache:mainfrom
Xuanwo:xuanwo/constructor-field-metadata

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Partially addresses #21982 for make_array, named_struct, and their shared argument coercion, building on #27536. Other constructors and aggregates remain in the issue's scope.

Rationale for this change

Even after argument types agree, constructor inference and execution can recreate fields from data types and lose their metadata. For example, make_array(named_struct('v', a), named_struct('v', b)) should retain the contributing fields' metadata through planning and execution.

What changes are included in this PR?

  • Preserve argument fields when their Arrow type is unchanged. When coercion changes the type, retain ordinary metadata and remove both extension metadata keys directly at the call sites (adding arrow-schema as a direct workspace dependency of datafusion-optimizer at the existing pinned Arrow 60.1.0 version, default-features = false), including during implicit literal conversion.
  • Have named_struct retain its value fields and make_array carry its inferred item field into execution.
  • Keep the public Spark array-adapter signature and respect its field name and offset width, including for empty inputs.

Shared Struct reconciliation and coercion error propagation landed in #27536 (04b910869) and are integrated from main, leaving only the constructor and scalar-argument coercion diff (7 files, +460/-37) in this PR.

What is the testing strategy for this PR?

Five SQL regressions in datafusion/core/tests/sql/select.rs check planned and executed types, metadata, values, and NULL validity across named_struct, make_array (including argument order, promotion, NULL elements, and nested Struct elements on top of #27536), scalar parameters, and conflicting extension annotations. Focused unit tests protect literal metadata presence (analyzer::type_coercion), argument-field coercion (type_coercion::functions), and the public and internal array adapters (make_array).

Integrated with main at 04b910869 (a81868b7d), this branch passes formatting, workspace all-target/all-feature clippy, focused checks (63 coercion tests, 42 analyzer tests, 163 nested-function tests, 26 SELECT tests, and 6 related SQLLogicTest files including coalesce.slt, struct.slt, metadata.slt, and array/make_array.slt), and the extended workspace suite (13,586 tests passed, 8 ignored, and 310 SQLLogicTest files completed).

Performance (historical stack measurement at e60efed)

Before #27536 merged and main advanced, the combined two-PR stack at e60efedf2a09d9ff9707a4e259577669f86ddb35 was compared against its then-base 120049489b5b74a3c8d942e5a97a1b7dbeec0ac2 using the same SQL harness, 16 batches of 4096 rows, Rust 1.99.1, release-nonlto, 12 pinned CPUs, serial ABBA order, and 30 samples per run. These figures are retained as historical measurements of that earlier stack (not rerun on a81868b7d after merging main at 04b910869, which advances upstream code and dependencies, and they do not isolate this PR from #27536):

Workload Base (120049489) Full stack (e60efed) Change
Same-type array planning 22.311 µs 23.328 µs +4.56%
Promoted array planning 23.825 µs 24.415 µs +2.48%
One-level Struct array planning 42.789 µs 44.394 µs +3.75%
Two-level Struct promotion planning 65.917 µs 68.786 µs +4.35%
Two-level Struct promotion, SQL-to-collect 5.157 ms 5.102 ms −1.05%

Historical constructor SQL-to-collect changes across regular and irregular batches ranged from −1.55% to +1.41%. The plain aggregate changed +3.49% and the Struct aggregate −0.06%; this is a correctness fix, with no overall speedup claim.

Are there any user-facing changes?

make_array and named_struct retain field metadata through planning and execution. Changed Arrow types lose their original extension annotations while retaining ordinary metadata. Building on #27536, nested Struct fields retain compatible metadata and nullability, and conflicting surviving extension names fail during planning.

@github-actions github-actions Bot added logical-expr Logical plan and expressions physical-expr Changes to the physical-expr crates optimizer Optimizer rules core Core DataFusion crate common Related to common crate functions Changes to functions implementation labels Nov 17, 2026
@github-actions github-actions Bot removed the physical-expr Changes to the physical-expr crates label Nov 17, 2026
@github-actions github-actions Bot removed the common Related to common crate label Nov 19, 2026
@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Nov 21, 2026
Xuanwo added a commit to Xuanwo/datafusion that referenced this pull request Nov 21, 2026
## Which issue does this PR close?

Related to apache#21982. This is the lower layer of the constructor metadata
work in apache#27502; it fixes shared Struct coercion independently of
constructor field propagation.

## Rationale for this change

Struct arguments can be assigned different child nullability or metadata
even when a function needs one common type. For example, COALESCE with a
non-nullable child in one argument and a nullable child in another can
fail with `Cannot cast nullable struct field 'v' to non-nullable field`.

## What changes are included in this PR?

Use one Struct reconciliation path behind the existing coercion entry
points. It preserves existing field-order choices, merges child
nullability and metadata by field name, ignores untyped NULL
contributors, and recurses into directly nested Structs. Changed child
types lose their extension annotations while retaining ordinary
metadata; conflicting surviving extension names produce a planning
error.

Route the existing array coercion adapter through that result so it
cannot swallow a metadata error and retry storage-only inference. Keep
its function-specific error for ordinary unification failures and its
public signature. Remove both extension metadata keys directly at the
call site, adding `arrow-schema` as a direct workspace dependency of
`datafusion-expr-common` at the existing pinned Arrow version (`60.1.0`,
`default-features = false`).

## What is the testing strategy for this PR?

SQL regressions over registered Struct columns check planned schemas and
executed arrays—types, metadata, values, and root/child NULL
validity—across COALESCE's simplified CASE path, a volatile expression
that retains native COALESCE execution, and `make_array`. They cover
extension preservation at unchanged child types, extension stripping on
child type promotion, ordinary metadata precedence, ignoring untyped
NULL-child metadata while widening child nullability, top-level and
nested field-order rules, and conflicting child extension planning
errors. SQLLogicTests cover pure-SQL Struct COALESCE coercion and
planning errors when comma-containing field names differ. A focused unit
test retains coverage for the public array-coercion adapter's
caller-named error contract.

The lower layer at `f03a6ec95` passed formatting, all-target/all-feature
clippy, 62 coercion tests, 162 nested-function tests, 20 SELECT tests,
`coalesce.slt`/`struct.slt`, and the extended workspace suite (13,453
tests passed, 8 ignored, and 307 SQLLogicTest files completed).

## Are there any user-facing changes?

Struct arguments resolve to one compatible type, including child
nullability and metadata. COALESCE accepts the nullable-child case
above. Conflicting surviving child extension names fail during planning
instead of being silently discarded by the array coercion fallback.
@github-actions github-actions Bot removed the sqllogictest SQL Logic Tests (.slt) label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation logical-expr Logical plan and expressions optimizer Optimizer rules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant