[SPARK-55630][SS] Skip updating matched flag for non-outer side in stream-stream join v4#54854
Open
HeartSaVioR wants to merge 1 commit intoapache:masterfrom
Open
[SPARK-55630][SS] Skip updating matched flag for non-outer side in stream-stream join v4#54854HeartSaVioR wants to merge 1 commit intoapache:masterfrom
HeartSaVioR wants to merge 1 commit intoapache:masterfrom
Conversation
HeartSaVioR
commented
Mar 17, 2026
| } | ||
| } | ||
|
|
||
| // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) |
Contributor
Author
There was a problem hiding this comment.
We can remove this test once we rebase after merging #54769
HeartSaVioR
commented
Mar 17, 2026
| } | ||
| } | ||
|
|
||
| // V1 excluded: V1 converter does not persist matched flags (SPARK-26154) |
Contributor
Author
There was a problem hiding this comment.
We can remove this test once we rebase after merging #54769
HeartSaVioR
force-pushed
the
SPARK-55630
branch
from
a86ec12 to
58ae81d
Compare
HeartSaVioR
force-pushed
the
SPARK-55630
branch
from
58ae81d to
cf3100c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR proposes to skip updating matched flag for non-outer side in stream-stream join v4.
Why are the changes needed?
After the insertion of the row into the state, we had to also "update" the row in the state back when it matches with the other side, to update the
matchedflag. This is not necessary for the join side which does not produce outer null; we want to avoid the unnecessary update.Note that the optimization is applicable to v2 and v3 as well. But this optimization would break the case where the user changes the join type during the restart (which is undocumented and arguably never guaranteed), so it'd be safer to only introduce the optimization in v4 only and apply back to v2/3 if you see demand.
Does this PR introduce any user-facing change?
No, the optimization is internal. We are introducing a behavioral change (despite undocumented and never guaranteed) but we only apply this optimization to v4, which is yet to be released.
How was this patch tested?
New UTs.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude 4.6 opus