[SPARK-52511][SDP] Support dry-run mode in spark-pipelines command #51489
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
python/pyspark/pipelines/cli.py
Outdated
| @@ -257,6 +257,13 @@ def run(spec_path: Path) -> None: | |||
| run_parser = subparsers.add_parser("run", help="Run a pipeline.") | |||
| run_parser.add_argument("--spec", help="Path to the pipeline spec.") | |||
|
|
|||
| # "dry-run" subcommand | |||
| run_parser = subparsers.add_parser( | |||
There was a problem hiding this comment.
@sryza shall we have an end-to-end test for the dry run mode? We should check that it can detect failures without side effects.
There was a problem hiding this comment.
Sure thing – just added, in test_spark_connect.py
gengliangwang
changed the title
gengliangwang
approved these changes
Jul 17, 2025
|
@sryza seems there is linter failure https://github.com/sryza/spark/actions/runs/16335212360/job/46145897514 |
python/pyspark/pipelines/cli.py
Outdated
| "dry-run", | ||
| help="Launch a run that just validates the graph and checks for errors.", | ||
| ) | ||
| run_parser.add_argument("--spec", help="Path to the pipeline spec.") |
There was a problem hiding this comment.
It seems to be added mistakenly. Please remove this duplication because we already have this at line 258, @sryza . 😄
There was a problem hiding this comment.
Thanks for catching – just fixed
|
cc @peter-toth |
dongjoon-hyun
approved these changes
Jul 17, 2025
|
Merged to master |
haoyangeng-db
pushed a commit
to haoyangeng-db/apache-spark
that referenced
this pull request
Jul 22, 2025
### What changes were proposed in this pull request? Adds a new `spark-pipelines` command that launches an execution of a pipeline that doesn't write or read any data, but catches many kinds of errors that would be caught if the pipeline were to actually run. E.g. - Syntax errors – e.g. invalid Python or SQL code - Analysis errors – e.g. selecting from a table that doesn't exist or selecting a column that doesn't exist - Graph validation errors - e.g. cyclic dependencies ### Why are the changes needed? Leverage the declarative nature of Declarative Pipelines to make pipeline development easier. ### Does this PR introduce _any_ user-facing change? Adds behavior; doesn't change existing behavior. ### How was this patch tested? - Added unit tests - Executed `dry-run` on the CLI, for both success and error scenarios ### Was this patch authored or co-authored using generative AI tooling? Closes apache#51489 from sryza/dry-run. Lead-authored-by: Sandy Ryza <[email protected]> Co-authored-by: Sandy Ryza <[email protected]> Signed-off-by: Sandy Ryza <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds a new
spark-pipelinescommand that launches an execution of a pipeline that doesn't write or read any data, but catches many kinds of errors that would be caught if the pipeline were to actually run. E.g.Why are the changes needed?
Leverage the declarative nature of Declarative Pipelines to make pipeline development easier.
Does this PR introduce any user-facing change?
Adds behavior; doesn't change existing behavior.
How was this patch tested?
dry-runon the CLI, for both success and error scenariosWas this patch authored or co-authored using generative AI tooling?