fix(schema): permit opt-in timestamp-precision evolution

[yihua]

Describe the issue this Pull Request addresses

Earlier Hudi versions mishandled long-backed timestamp logical types in AvroInternalSchemaConverter:

1. timestamp-millis and timestamp-micros both collapsed into a single internal TimestampType and were always re-emitted as timestampMicros() on serialize. A source schema declaring timestamp-millis got persisted in the table with the wrong timestamp-micros logical type, while the underlying long values written to parquet remained epoch-millis. Pure logical-type drift.
2. local-timestamp-millis and local-timestamp-micros had no branch at all. They fell through to the bare LongType, and the logical type was dropped from the table schema entirely. Logical-type loss.

In both cases the parquet values are correct; only the logical type on the field is wrong. Current converters recognize all four logical types as distinct, so the writer schema now declares the correct logical type. On every subsequent write the reconcile path compares writer schema against the persisted table schema, finds the logical-type mismatch, and rejects it.

With hoodie.write.set.null.for.missing.columns=false the table schema already self-repairs on the next commit: HoodieSchemaUtils.deduceWriterSchema skips reconcileSchema entirely and lets AvroSchemaCompatibility.checkReaderWriterCompatibility validate. That check is logical-type-blind (both timestamps are long underneath), so it accepts the corrected logical type from the writer schema and the next commit rewrites the table schema's logical type accordingly. No change is needed for this path.

With hoodie.write.set.null.for.missing.columns=true the repair is blocked. HoodieSchemaUtils.deduceWriterSchema instead calls AvroSchemaEvolutionUtils.reconcileSchema, which goes through TableChanges.ColumnUpdateChange.updateColumnType and SchemaChangeUtils.isTypeUpdateAllow. That switch had no case for TIMESTAMP or TIMESTAMP_MILLIS, so any logical-type change fell into default: return false and threw SchemaCompatibilityException. The reconcile path was strictly stricter than the non-reconcile path for the same scenario; this PR fixes only that one path.

Complements the read-side repair from #14161, which handles parquet files carrying the wrong logical type transparently. This PR closes the write-side gap so the table schema itself can be brought into agreement with the writer schema even when set.null.for.missing.columns=true.

Summary and Changelog

Users gain a per-write opt-in to forward-fix tables whose persisted schema carries a wrong or missing timestamp logical type, by allowing the internal-schema reconcile path to update the column's logical type to match the writer schema. The non-reconcile path (set.null.for.missing.columns=false) already repaired the logical type implicitly via the Avro reader/writer compatibility check; this PR brings the reconcile path to parity. Default behavior is unchanged.

Changes:

• New advanced write config hoodie.write.schema.allow.timestamp.precision.evolution on HoodieCommonConfig (default false, sinceVersion("1.3.0")).
• SchemaChangeUtils.isTypeUpdateAllow gains a boolean allowTimestampPrecisionEvolution parameter. When true, the switch permits:
  • timestamp-millis ↔ timestamp-micros (logical-type drift case, both directions)
  • local-timestamp-millis ↔ local-timestamp-micros (precision swap among the recognized variants)
  • long → local-timestamp-millis / long → local-timestamp-micros (logical-type loss case, attach the missing logical type)
• AvroSchemaEvolutionUtils.reconcileSchema gains an overload that threads the flag through to TableChanges.ColumnUpdateChange, which stores it on the change and passes it to isTypeUpdateAllow. Pre-existing reconcileSchema / ColumnUpdateChange.get overloads kept as delegates.
• The four writer-side callers (HoodieSchemaUtils.scala, BaseHoodieWriteClient, HoodieMergeHelper, FileGroupReaderBasedMergeHandle) read the config from the write properties and pass it through to reconcileSchema.
• Tests:
  • TestAvroSchemaEvolutionUtils.testReconcileSchemaTimestampPrecisionEvolution asserts default-strict reject and opt-in permit for all three permitted shapes.
  • testCOWLogicalRepair / testMORLogicalRepair parameterize on both setNullForMissingColumns and allowTimestampPrecisionEvolution; positive variants exercise the gated repair path on the v6/v8/CURRENT logical-repair fixtures from fix(ingest): Repair affected logical timestamp milli tables #14161; a negative variant asserts SchemaCompatibilityException when the reconcile path is on with the gate closed.

Impact

• New advanced write config, opt-in. Default preserves the prior strict behavior; no existing caller sees a change.
• One new overload on AvroSchemaEvolutionUtils.reconcileSchema, one new factory on TableChanges.ColumnUpdateChange.get. Pre-existing overloads kept as delegates, so no public-API breakage.

Risk Level

low

The gate defaults off, so existing writers are unaffected. Test coverage adds positive variants on the existing logical-repair fixtures and a negative variant locking in the default-strict behavior.

Documentation Update

New config documented inline on HoodieCommonConfig.ALLOW_TIMESTAMP_PRECISION_EVOLUTION with sinceVersion("1.3.0"). No website update needed.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[hudi-bot]

CI report:

• dc16a7a UNKNOWN
• d4c48a7 UNKNOWN
• 176eb56 Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[wombatu-kun]

This omits the third behavior the flag gates: attaching a logical type to a bare long (long -> local-timestamp-millis/micros), the logical-type-loss repair the PR description calls a primary motivation. A user whose 0.x table stored the column as bare long would not learn from "precision-only evolution between these logical types" that this flag applies. Suggest documenting the long -> local-timestamp attach case explicitly.

[yihua]

long -> local-timestamp is backward compatible for readers. The logical type local-timestamp is now supported on master, so we can keep this part out for simplicity.