Repository navigation
fix: reconcile Struct fields during type coercion - #27536
Conversation
| /// | ||
| /// [API policy]: https://datafusion.apache.org/contributor-guide/api-health.html#internal-public-apis | ||
| #[doc(hidden)] | ||
| pub fn strip_extension_type(metadata: &mut Metadata) { |
There was a problem hiding this comment.
Thinking if this new pub interface is necessary, can we remove those metadata directly on where this util method is used? Maybe we need to add a new dep arrow-schema, but it's indirect dep of arrow anyway
There was a problem hiding this comment.
Removed strip_extension_type and now remove both extension metadata keys directly at the call sites. Added arrow-schema as a direct workspace dependency where needed; the pinned Arrow version and features are unchanged.
| } | ||
|
|
||
| #[test] | ||
| fn test_struct_type_union_field_contracts() -> Result<()> { |
There was a problem hiding this comment.
I think this would be much better to test with .slt / SQL / dataframe as that is the end user API -- these tests don't really pin the behavior from a user perspective
I realize we may have to make a table with a registered type
There was a problem hiding this comment.
Thanks! Moved these to SQL tests over registered StructArray columns, and added slt cases for the field-name errors. The old coercion unit tests are removed.
Which issue does this PR close?
Related to #21982. This is the lower layer of the constructor metadata work in #27502; it fixes shared Struct coercion independently of constructor field propagation.
Rationale for this change
Struct arguments can be assigned different child nullability or metadata even when a function needs one common type. For example, COALESCE with a non-nullable child in one argument and a nullable child in another can fail with
Cannot cast nullable struct field 'v' to non-nullable field.What changes are included in this PR?
Use one Struct reconciliation path behind the existing coercion entry points. It preserves existing field-order choices, merges child nullability and metadata by field name, ignores untyped NULL contributors, and recurses into directly nested Structs. Changed child types lose their extension annotations while retaining ordinary metadata; conflicting surviving extension names produce a planning error.
Route the existing array coercion adapter through that result so it cannot swallow a metadata error and retry storage-only inference. Keep its function-specific error for ordinary unification failures and its public signature. Remove both extension metadata keys directly at the call site, adding
arrow-schemaas a direct workspace dependency ofdatafusion-expr-commonat the existing pinned Arrow version (60.1.0,default-features = false).What is the testing strategy for this PR?
SQL regressions over registered Struct columns check planned schemas and executed arrays—types, metadata, values, and root/child NULL validity—across COALESCE's simplified CASE path, a volatile expression that retains native COALESCE execution, and
make_array. They cover extension preservation at unchanged child types, extension stripping on child type promotion, ordinary metadata precedence, ignoring untyped NULL-child metadata while widening child nullability, top-level and nested field-order rules, and conflicting child extension planning errors. SQLLogicTests cover pure-SQL Struct COALESCE coercion and planning errors when comma-containing field names differ. A focused unit test retains coverage for the public array-coercion adapter's caller-named error contract.The lower layer at
f03a6ec95passed formatting, all-target/all-feature clippy, 62 coercion tests, 162 nested-function tests, 20 SELECT tests,coalesce.slt/struct.slt, and the extended workspace suite (13,453 tests passed, 8 ignored, and 307 SQLLogicTest files completed).Are there any user-facing changes?
Struct arguments resolve to one compatible type, including child nullability and metadata. COALESCE accepts the nullable-child case above. Conflicting surviving child extension names fail during planning instead of being silently discarded by the array coercion fallback.