refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs - #5197
Open
andygrove wants to merge 1 commit into
Open
refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs#5197andygrove wants to merge 1 commit into
andygrove wants to merge 1 commit into
Conversation
… docs Standardizes the acronym casing in `spark.comet.exec.pyarrowUdf.enabled` -> `spark.comet.exec.pyarrowUDF.enabled`, matching the existing `spark.comet.exec.scalaUDF.codegen.enabled` and the acronym rule in the configuration conventions guide. This key was added in apache#4234 and has not appeared in a release, so no `withAlternative` alias is needed. The Scala symbol (`COMET_PYARROW_UDF_ENABLED`) was already correct and is unchanged. Also sweeps `config_conventions.md` for references that no longer match the code: - `columnar` is no longer a category segment (the shuffle configs were unified under `spark.comet.shuffle.*` in apache#4986); added `debug`, which was missing. - The symbol-naming table listed `COMET_ROW_FILTER_PUSHDOWN`; the actual symbol is `COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED`. - The acronym rule cited `pyarrowUdf` as a live example of mixed casing, which this commit fixes. - The behavior-change section instructs contributors to use `.category(CATEGORY_LEGACY)`, but no legacy config or `CATEGORY_LEGACY` constant exists yet; note that the first one to land must add it. Part of apache#4978.
10 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Part of #4978.
Rationale for this change
spark.comet.exec.pyarrowUdf.enabledandspark.comet.exec.scalaUDF.codegen.enabledspell the same acronym two different ways. The configuration conventions guide calls this out by name as a red flag, so the guide was citing a live inconsistency in our own config surface.The key was added in #4234 and has not shipped in a release (it is not in 0.17.1), so it can be renamed outright with no deprecated alias — which is only true until 1.0 goes out. Doing it now costs nothing; doing it later means carrying a
withAlternativethrough all of 1.x.While in the conventions doc I swept it for other references that no longer match the code.
What changes are included in this PR?
Rename (no
withAlternative, key is unreleased):spark.comet.exec.pyarrowUdf.enabled→spark.comet.exec.pyarrowUDF.enabledThe Scala symbol
COMET_PYARROW_UDF_ENABLEDwas already correct, so all Scala callsites andCometMapInBatchSuiteare untouched. Only the key string changes, inCometConf.scala, one doc comment, the user guide, and the two pyspark scripts that set the key by name.Stale references fixed in
config_conventions.md:columnaris a category segmentspark.comet.columnar.*key remains — unified underspark.comet.shuffle.*in #4986debugabsent from the category list)spark.comet.debug.enabledexistsspark.comet.parquet.rowFilterPushdown.enabled→COMET_ROW_FILTER_PUSHDOWNCOMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLEDpyarrowUdfas mixed-casing example.category(CATEGORY_LEGACY)CATEGORY_LEGACYconstant exists yet; now notes the first one to land must add itHow are these changes tested?
mvnw compile -Pspark-4.0(the profile that owns the PyArrow UDF path) is clean, as isspotless:check. The rename is a pure key-string change with no behavior change; existingCometMapInBatchSuitecoverage goes through the Scala symbol and is unaffected.