Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/rust.yml
Original file line number Diff line number Diff line change
Expand Up @@ -706,6 +706,7 @@ jobs:
run: |
cargo xtask ci step test substrait
cargo xtask ci step test substrait-optimized
cargo xtask ci step test unparser

# Temporarily commenting out the Windows flow, the reason is enormously slow running build
# Waiting for new Windows 2025 github runner
Expand Down
42 changes: 41 additions & 1 deletion datafusion/sqllogictest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -259,7 +259,8 @@ Runs the same `.slt` once per combination of config values. Each directive is a
- Each override is applied like `SET <key> = <value>`, so `datafusion.runtime.*`
keys (e.g. `datafusion.runtime.memory_limit`) and config-dependent UDFs behave
exactly as they would for an in-file `SET`.
- Supported by the default runner and by `--substrait-round-trip`. `--complete`
- Supported by the default runner, by `--substrait-round-trip`, and by
`--unparser-round-trip`. `--complete`
rejects a file that declares directives, since it would overwrite the file
with the output of a single combination. The Postgres runner ignores them.
- Every combination runs even if an earlier one fails. Each failure is prefixed
Expand Down Expand Up @@ -487,6 +488,45 @@ and uncorrelated `WHERE EXISTS`:
cargo test --test sqllogictests --features substrait -- --substrait-round-trip --substrait-optimize joins_conditionless.slt
```

## Running tests: Unparser round-trip mode

This mode runs each query through the SQL unparser. The runner plans the SQL, unparses the
logical plan with `plan_to_sql` and the default dialect, plans the unparsed SQL again in the
same `SessionContext`, and executes the result (SQL statement → DF logical → SQL statement →
DF logical → DF physical → execute).

The runner passes `Ddl`, `Dml`, `Copy`, `Explain`, `Analyze`, `Show` and `Statement` plans through unchanged,
because SQL cannot express them as a query.

| Failure | Meaning |
| ----------------------- | ---------------------------------------------- |
| Error while unparsing | The unparser does not support the plan. |
| Error while re-planning | The unparser produced SQL that does not plan. |
| Different rows | The unparser changed the meaning of the query. |

Many files fail in this mode. [`unparser_known_failures.txt`](unparser_known_failures.txt) lists them, one path
relative to `test_files` per line. CI runs every other file through `cargo xtask ci step test unparser`:

```shell
cargo test --test sqllogictests -- --unparser-round-trip --unparser-known-failures unparser_known_failures.txt
```

A new `.slt` file runs in this mode by default. If the unparser cannot handle it yet, add the file to the list.
When a fix makes a listed file pass, remove the file from the list. The runner rejects an entry that names no
file in `test_files`.

To run every file, including the listed ones, leave out `--unparser-known-failures`:

```shell
cargo test --test sqllogictests -- --unparser-round-trip
```

Add `--unparser-optimize` to optimize the logical plan before unparsing it:

```shell
cargo test --test sqllogictests -- --unparser-round-trip --unparser-optimize select.slt
```

## `.slt` file format

[`sqllogictest`] was originally written for SQLite to verify the
Expand Down
110 changes: 105 additions & 5 deletions datafusion/sqllogictest/bin/sqllogictests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ use datafusion::common::{DataFusionError, Result, exec_datafusion_err, exec_err}
use datafusion::execution::memory_pool::DEFAULT_DRIFT_LOG_THRESHOLD;
#[cfg(feature = "substrait")]
use datafusion_sqllogictest::DataFusionSubstraitRoundTrip;
use datafusion_sqllogictest::DataFusionUnparserRoundTrip;
use datafusion_sqllogictest::TestFile;
use datafusion_sqllogictest::{
CountingAllocator, enable_memory_drift_logging, flush_thread_allocations,
Expand Down Expand Up @@ -37,6 +38,7 @@ use crate::postgres_container::{
};
use datafusion::common::runtime::SpawnedTask;
use futures::FutureExt;
use std::collections::HashSet;
use std::fmt::Write as _;
use std::fs;
use std::io::{IsTerminal, Write, stderr, stdout};
Expand Down Expand Up @@ -216,8 +218,24 @@ async fn run_tests() -> Result<()> {
options.postgres_runner,
options.complete,
options.substrait_round_trip,
options.unparser_round_trip,
) {
(_, _, true) => {
(_, _, _, true) => {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be nice to refactor into a struct

run_matrix(
Engine::UnparserRoundTrip {
optimize: options.unparser_optimize,
},
test_file,
validator,
m_clone,
m_style_clone,
filters.as_ref(),
currently_running_sql_tracker_clone,
colored_output,
)
.await
}
(_, _, true, _) => {
run_test_file_substrait_round_trip(
test_file,
validator,
Expand All @@ -230,7 +248,7 @@ async fn run_tests() -> Result<()> {
)
.await
}
(false, false, _) => {
(false, false, _, _) => {
run_test_file(
test_file,
validator,
Expand All @@ -242,7 +260,7 @@ async fn run_tests() -> Result<()> {
)
.await
}
(false, true, _) => {
(false, true, _, _) => {
run_complete_file(
test_file,
validator,
Expand All @@ -252,7 +270,7 @@ async fn run_tests() -> Result<()> {
)
.await
}
(true, false, _) => {
(true, false, _, _) => {
run_test_file_with_postgres(
test_file,
validator,
Expand All @@ -263,7 +281,7 @@ async fn run_tests() -> Result<()> {
)
.await
}
(true, true, _) => {
(true, true, _, _) => {
run_complete_file_with_postgres(
test_file,
validator,
Expand Down Expand Up @@ -452,6 +470,9 @@ enum Engine {
SubstraitRoundTrip {
optimize: bool,
},
UnparserRoundTrip {
optimize: bool,
},
}

impl Engine {
Expand All @@ -462,6 +483,7 @@ impl Engine {
Engine::DataFusion => "Datafusion",
#[cfg(feature = "substrait")]
Engine::SubstraitRoundTrip { .. } => "DatafusionSubstraitRoundTrip",
Engine::UnparserRoundTrip { .. } => "DatafusionUnparserRoundTrip",
}
}
}
Expand Down Expand Up @@ -620,6 +642,9 @@ impl MatrixRunner<'_> {
Engine::SubstraitRoundTrip { optimize } => {
self.run_substrait_round_trip(&test_ctx, pb, optimize).await
}
Engine::UnparserRoundTrip { optimize } => {
self.run_unparser_round_trip(&test_ctx, pb, optimize).await
}
}
}

Expand Down Expand Up @@ -702,6 +727,28 @@ impl MatrixRunner<'_> {
self.drive_runner(&mut runner, &pb).await
}

/// Unparser round-trip engine: each query's logical plan is unparsed to SQL
/// and the SQL is planned again before execution.
async fn run_unparser_round_trip(
&self,
test_ctx: &TestContext,
pb: ProgressBar,
optimize: bool,
) -> Result<()> {
let mut runner = sqllogictest::Runner::new(|| async {
Ok(DataFusionUnparserRoundTrip::new(
test_ctx.session_ctx().clone(),
self.relative_path.to_path_buf(),
pb.clone(),
)
.with_optimization(optimize)
.with_currently_executing_sql_tracker(
self.currently_executing_sql_tracker.clone(),
))
});
self.drive_runner(&mut runner, &pb).await
}

/// Label the runner, install the shared validators, replay the records, and
/// clear the progress bar. Engine-independent, so every engine shares it.
async fn drive_runner<D, M>(
Expand Down Expand Up @@ -1033,10 +1080,39 @@ fn read_test_files(options: &Options) -> Result<Vec<TestFile>> {
.filter(|f| options.check_pg_compat_file(f.path.as_path()))
.collect::<Vec<_>>();

if let Some(list) = &options.unparser_known_failures {
let known_failures = read_known_failures(list)?;
// A stale entry (renamed or deleted file) would otherwise go unnoticed.
if let Some(missing) = known_failures
.iter()
.find(|p| !Path::new(TEST_DIRECTORY).join(p).is_file())
{
return exec_err!(
"{} lists {}, which is not a file in {TEST_DIRECTORY}",
list.display(),
missing.display()
);
}
paths.retain(|f| !known_failures.contains(&f.relative_path));
}

paths.sort_unstable();
Ok(paths)
}

/// Reads a known-failures list: one path relative to `test_files` per line.
/// Blank lines and lines that start with `#` are ignored.
fn read_known_failures(list: &Path) -> Result<HashSet<PathBuf>> {
let content = fs::read_to_string(list)
.map_err(|e| exec_datafusion_err!("Error reading {}: {e}", list.display()))?;
Ok(content
.lines()
.map(str::trim)
.filter(|line| !line.is_empty() && !line.starts_with('#'))
.map(PathBuf::from)
.collect())
}

/// Parsed command line options
///
/// This structure attempts to mimic the command line options of the built-in rust test runner
Expand Down Expand Up @@ -1071,6 +1147,30 @@ struct Options {
)]
substrait_optimize: bool,

#[clap(
long,
conflicts_with = "complete",
conflicts_with = "postgres_runner",
conflicts_with = "substrait_round_trip",
help = "Before executing each query, unparse its logical plan to SQL and plan that SQL again"
)]
unparser_round_trip: bool,

#[clap(
long,
requires = "unparser_round_trip",
help = "Optimize logical plans before unparsing them in unparser round-trip mode"
)]
unparser_optimize: bool,

#[clap(
long,
value_name = "FILE",
requires = "unparser_round_trip",
help = "Skip the .slt files listed in FILE, one path relative to test_files per line"
)]
unparser_known_failures: Option<PathBuf>,

#[clap(long, env = "INCLUDE_SQLITE", help = "Include sqlite files")]
include_sqlite: bool,

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ use std::{path::PathBuf, time::Duration};
use crate::engines::currently_executed_sql::CurrentlyExecutingSqlTracker;
use crate::engines::datafusion_engine::Result;
use crate::engines::output::{DFColumnType, DFOutput};
use crate::engines::skips_round_trip;
use crate::memory_drift::rewrap_replaced_pool;
use crate::{DFSqlLogicTestError, convert_batches, convert_schema_to_types};
use arrow::record_batch::RecordBatch;
use async_trait::async_trait;
use datafusion::logical_expr::LogicalPlan;
use datafusion::physical_plan::common::collect;
use datafusion::physical_plan::execute_stream;
use datafusion::prelude::SessionContext;
Expand Down Expand Up @@ -154,19 +154,11 @@ async fn run_query_substrait_round_trip(
} else {
df.into_unoptimized_plan()
};
let round_tripped_plan = match &logical_plan {
// Substrait does not handle these plans
LogicalPlan::Ddl(_)
| LogicalPlan::Explain(_)
| LogicalPlan::Dml(_)
| LogicalPlan::Copy(_)
| LogicalPlan::Show(_)
| LogicalPlan::Statement(_) => logical_plan,
// For any other plan, convert to Substrait
logical_plan => {
let plan = to_substrait_plan(logical_plan, &state)?;
from_substrait_plan(&state, &plan).await?
}
let round_tripped_plan = if skips_round_trip(&logical_plan) {
logical_plan
} else {
let plan = to_substrait_plan(&logical_plan, &state)?;
from_substrait_plan(&state, &plan).await?
};

let physical_plan = state.create_physical_plan(&round_tripped_plan).await?;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements

mod runner;

pub use runner::*;
Loading