Skip to content

[core] Fix schema evolution for multiset elements - #9545

Open
ArnavBalyan wants to merge 2 commits into
apache:masterfrom
ArnavBalyan:arnavb/fix-ms
Open

[core] Fix schema evolution for multiset elements#9545
ArnavBalyan wants to merge 2 commits into
apache:masterfrom
ArnavBalyan:arnavb/fix-ms

Conversation

@ArnavBalyan

Copy link
Copy Markdown
Member

Purpose

  • Schema evolution fails when updating nested fields or multiset elements.
  • Paimon generates the nested schema, but the schema manager does not traverse multiset elements and reports that the element column does not exist.
  • Add multiset traversal/reconstruction to apply nested field evolution.

Tests

  • UT

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Support reading existing multiset data after committing the schema change

This change lets SchemaManager commit a multiset element-type evolution, but the read-side schema-evolution path still has no MultisetType support. SchemaEvolutionUtil.createCastExecutor handles RowType, ArrayType, and MapType; a multiset change falls through to CastExecutors.resolve, which returns null for multiset-to-multiset casts.

I reproduced this on the current PR head with the following sequence:

  1. Create a table containing MULTISET<INT>.
  2. Write a row under that schema.
  3. Alter the element type to BIGINT.
  4. Read the existing row.

The ALTER succeeds, but the read fails with:

NullPointerException: Cannot cast from type MULTISET<INT> to type MULTISET<BIGINT>
    at SchemaEvolutionUtil.createCastExecutor(SchemaEvolutionUtil.java:262)

The same missing read path also affects nested row evolution inside a multiset, such as adding a field or widening an existing field, whenever files written with the old schema are present.

Please add a MultisetType branch to the read-side evolution logic. Since a multiset is represented as an InternalMap, it should recursively cast the key array (the elements) while preserving the integer multiplicity value array. An end-to-end regression test should write data before the schema change and read it afterward; the current tests only verify the resulting metadata.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED. Implementation: FINDINGS.

The earlier missing read-side multiset cast is fixed: the new write-before-ALTER/read-afterward regression passes for INT to BIGINT and nested row widening/addition. There is still a multiplicity-loss case when an accepted element evolution maps distinct old elements to equal new elements; details are inline.

Validation: 60 SchemaManager/table tests passed after compiling this PR's core sources and matching RESTApi, avoiding the newer cached SchemaManager-interface ABI. An additional real stored-table probe checked three cases: INT to BIGINT preserved total multiplicity 5; INT to FLOAT with colliding values, and dropping a distinguishing field from a multiset's row element, both produced duplicate keys and total multiplicity 3 after the standard row-value copy. This was a local core read/copy reproduction, not a Flink or Spark SQL run.

createCastExecutor(
inputType.getElementType(), targetType.getElementType()));

CastedMap castedMap = CastedMap.fromKey(castElementGetter);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Preserve multiplicities when evolved elements become equal

The new key-only lazy cast keeps one entry per old element even when an allowed schema change makes their keys equal. I reproduced this with real table writes followed by ALTER: MULTISET<INT> containing {16777216:2, 16777217:3} can evolve to FLOAT, yielding two identical float keys; InternalRowUtils.copy then overwrites the first count and returns multiplicity 3 instead of 5. Dropping a from MULTISET<ROW<a INT,b INT>> containing {(10,1):2,(20,1):3} has the same failure. INT to BIGINT is a passing control. These changes are accepted by the current schema manager, so returning duplicate keys violates the multiset representation and loses occurrences through normal materialization. Either combine multiplicities for equal evolved elements, or reject element evolutions that cannot preserve uniqueness before committing the schema. Please add write/ALTER/read-and-copy regressions for these collisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants