Skip to content

fix: reconcile Struct fields during type coercion - #27536

Merged
alamb merged 4 commits into
apache:mainfrom
Xuanwo:xuanwo/struct-field-coercion
Nov 21, 2026
Merged

alamb merged 4 commits into
apache:mainfrom
Xuanwo:xuanwo/struct-field-coercion

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Which issue does this PR close?

Related to #21982. This is the lower layer of the constructor metadata work in #27502; it fixes shared Struct coercion independently of constructor field propagation.

Rationale for this change

Struct arguments can be assigned different child nullability or metadata even when a function needs one common type. For example, COALESCE with a non-nullable child in one argument and a nullable child in another can fail with Cannot cast nullable struct field 'v' to non-nullable field.

What changes are included in this PR?

Use one Struct reconciliation path behind the existing coercion entry points. It preserves existing field-order choices, merges child nullability and metadata by field name, ignores untyped NULL contributors, and recurses into directly nested Structs. Changed child types lose their extension annotations while retaining ordinary metadata; conflicting surviving extension names produce a planning error.

Route the existing array coercion adapter through that result so it cannot swallow a metadata error and retry storage-only inference. Keep its function-specific error for ordinary unification failures and its public signature. Remove both extension metadata keys directly at the call site, adding arrow-schema as a direct workspace dependency of datafusion-expr-common at the existing pinned Arrow version (60.1.0, default-features = false).

What is the testing strategy for this PR?

SQL regressions over registered Struct columns check planned schemas and executed arrays—types, metadata, values, and root/child NULL validity—across COALESCE's simplified CASE path, a volatile expression that retains native COALESCE execution, and make_array. They cover extension preservation at unchanged child types, extension stripping on child type promotion, ordinary metadata precedence, ignoring untyped NULL-child metadata while widening child nullability, top-level and nested field-order rules, and conflicting child extension planning errors. SQLLogicTests cover pure-SQL Struct COALESCE coercion and planning errors when comma-containing field names differ. A focused unit test retains coverage for the public array-coercion adapter's caller-named error contract.

The lower layer at f03a6ec95 passed formatting, all-target/all-feature clippy, 62 coercion tests, 162 nested-function tests, 20 SELECT tests, coalesce.slt/struct.slt, and the extended workspace suite (13,453 tests passed, 8 ignored, and 307 SQLLogicTest files completed).

Are there any user-facing changes?

Struct arguments resolve to one compatible type, including child nullability and metadata. COALESCE accepts the nullable-child case above. Conflicting surviving child extension names fail during planning instead of being silently discarded by the array coercion fallback.

@github-actions github-actions Bot added logical-expr Logical plan and expressions core Core DataFusion crate common Related to common crate functions Changes to functions implementation labels Nov 17, 2026

@waynexia waynexia left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest LGTM, thanks

Comment thread datafusion/common/src/metadata.rs Outdated
///
/// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis
#[doc(hidden)]
pub fn strip_extension_type(metadata: &mut Metadata) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking if this new pub interface is necessary, can we remove those metadata directly on where this util method is used? Maybe we need to add a new dep arrow-schema, but it's indirect dep of arrow anyway

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed strip_extension_type and now remove both extension metadata keys directly at the call sites. Added arrow-schema as a direct workspace dependency where needed; the pinned Arrow version and features are unchanged.

@github-actions github-actions Bot removed the common Related to common crate label Nov 19, 2026

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Xuanwo and @waynexia

}

#[test]
fn test_struct_type_union_field_contracts() -> Result<()> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this would be much better to test with .slt / SQL / dataframe as that is the end user API -- these tests don't really pin the behavior from a user perspective

I realize we may have to make a table with a registered type

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Moved these to SQL tests over registered StructArray columns, and added slt cases for the field-name errors. The old coercion unit tests are removed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you

@github-actions github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Nov 21, 2026
@alamb
alamb added this pull request to the merge queue Nov 21, 2026
Merged via the queue into apache:main with commit 04b9108 Nov 21, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate functions Changes to functions implementation logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) v57.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants