[SPARK-56045][SQL] Add flag for ignoring Parquet UNKNOWN type annotation and revert to old behavior

[ZiyaZa]

What changes were proposed in this pull request?

This PR introduces a new flag spark.sql.parquet.reader.respectUnknownTypeAnnotation.enabled for Parquet reader to control the behavior when it reads an external file with UNKNOWN logical type annotation:

• (Default) When false, we infer the Spark type based on the physical type used in the Parquet file, as we did before Spark 4.1.
• When true, we use NullType as the Spark type.

Why are the changes needed?

To fix the regression introduced by #52922, as we have been reading files differently since then.

Does this PR introduce any user-facing change?

Yes. With default flag value, when we read a Parquet file written by an external engine:

• Before, we inferred NullType
• Now, we'll infer a type based on the physical type (e.g. IntegerType)

How was this patch tested?

Added tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[cloud-fan]

LGTM if CI is green, please create a new JIRA ticket as the original commit is already released.

[ZiyaZa]

LGTM if CI is green, please create a new JIRA ticket as the original commit is already released.

CI is green, linked the new ticket in the title.