Skip to content

chore: Reenable nested types for CometFuzzTestSuite with int96 #1761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
May 21, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion docs/source/user-guide/compatibility.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ logical types. Arrow-based readers, such as DataFusion and Comet do respect thes
rather than signed. By default, Comet will fall back to Spark when scanning Parquet files containing `byte` or `short`
types (regardless of the logical type). This behavior can be disabled by setting
`spark.comet.scan.allowIncompatible=true`.
- Reading legacy INT96 timestamps contained within complex types can produce different results to Spark
- There is a known performance issue when pushing filters down to Parquet. See the [Comet Tuning Guide] for more
information.
- There are failures in the Spark SQL test suite when enabling these new scans (tracking issues: [#1542] and [#1545]).
Expand Down
1 change: 0 additions & 1 deletion docs/templates/compatibility-template.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,6 @@ The new scans currently have the following limitations:
rather than signed. By default, Comet will fall back to Spark when scanning Parquet files containing `byte` or `short`
types (regardless of the logical type). This behavior can be disabled by setting
`spark.comet.scan.allowIncompatible=true`.
- Reading legacy INT96 timestamps contained within complex types can produce different results to Spark
- There is a known performance issue when pushing filters down to Parquet. See the [Comet Tuning Guide] for more
information.
- There are failures in the Spark SQL test suite when enabling these new scans (tracking issues: [#1542] and [#1545]).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -247,12 +247,7 @@ class CometFuzzTestSuite extends CometTestBase with AdaptiveSparkPlanHelper {
}

test("Parquet temporal types written as INT96") {
// int96 coercion in DF does not work with nested types yet
// https://github.com/apache/datafusion/issues/15763
testParquetTemporalTypes(
ParquetOutputTimestampType.INT96,
generateArray = false,
generateStruct = false)
testParquetTemporalTypes(ParquetOutputTimestampType.INT96)
}

test("Parquet temporal types written as TIMESTAMP_MICROS") {
Expand Down
Loading