fix: Fallback to Spark when PARQUET_FIELD_ID_READ_ENABLED=true for new native scans

[andygrove]

Which issue does this PR close?

Part of #1758

Rationale for this change

Fix some Spark SQL test failures when the new native scans are enabled

What changes are included in this PR?

How are these changes tested?

I manually tested with the following test:

testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetFieldIdIOSuite -- -z "read parquet file without ids"

[andygrove]

This is the change. The rest is refactoring.

[andygrove]

This may make too many tests fall back because Spark may be enabling this by default in all tests ... investigating

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 65.21739% with 8 lines in your changes missing coverage. Please review.

Project coverage is 58.56%. Comparing base (f09f8af) to head (4a84257).
Report is 199 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/scala/org/apache/comet/rules/CometScanRule.scala	65.21%	3 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1757      +/-   ##
============================================
+ Coverage     56.12%   58.56%   +2.43%     
- Complexity      976     1133     +157     
============================================
  Files           119      130      +11     
  Lines         11743    12686     +943     
  Branches       2251     2369     +118     
============================================
+ Hits           6591     7429     +838     
- Misses         4012     4065      +53     
- Partials       1140     1192      +52     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[parthchandra]

Pre-emptively approving this. We can defer field_id support for native readers for the time being.

[andygrove]

SQLConf.PARQUET_FIELD_ID_READ_ENABLED is enabled in all Spark tests, so not sure what to do about this now.

private[sql] object TestSQLContext {

  /**
   * A map used to store all confs that need to be overridden in sql/core unit tests.
   */
  val overrideConfs: Map[String, String] =
    Map(
      // Fewer shuffle partitions to speed up testing.
      SQLConf.SHUFFLE_PARTITIONS.key -> "5",
      // Enable parquet read field id for tests to ensure correctness
      // By default, if Spark schema doesn't contain the `parquet.field.id` metadata,
      // the underlying matching mechanism should behave exactly like name matching
      // which is the existing behavior. Therefore, turning this on ensures that we didn't
      // introduce any regression for such mixed matching mode.
      SQLConf.PARQUET_FIELD_ID_READ_ENABLED.key -> "true")
}

[parthchandra]

Ouch.

[comphead]

Suggested change

	withInfo(plan, s"Comet $scanImpl scan does not support PARQUET_FIELD_ID_READ_ENABLED")
	withInfo(plan, s"Comet $scanImpl scan does not support with enabled `spark.sql.parquet.fieldId.read.enabled`")

[comphead]

lgtm thanks @andygrove

[andygrove]

I will close this since this will make all Spark SQL tests fall back to Spark. We seem to mostly support this feature in native_iceberg_compat already. I guess we have no choice but to add support in native_datafusion.