Optimize splitPart(limit) to avoid full array allocation in hot query path#19049
Open
Akanksha-kedia wants to merge 1 commit into
Open
Optimize splitPart(limit) to avoid full array allocation in hot query path#19049Akanksha-kedia wants to merge 1 commit into
Akanksha-kedia wants to merge 1 commit into
Conversation
The 4-argument splitPart overload previously allocated a full String[] via StringUtils.splitByWholeSeparator on every call, even when only a single element was needed. This is wasteful in hot query paths where splitPart is invoked per-row. Replace the array-based implementation with index-based forward scanning that extracts only the requested field without materializing all split parts. The new implementation: - Skips leading separators - Collapses consecutive separators (matching splitByWholeSeparator semantics) - Handles trailing separators (producing one empty trailing token) - For positive indices: single forward scan, O(index) work - For negative indices: two passes (count then extract) but still no String[] allocation Falls back to the array-based path only for null/empty delimiters (whitespace splitting) where the rules are complex. All existing unit tests (172 cases including randomized fuzz) pass unchanged, confirming behavioral equivalence.
Contributor
Author
|
cc @walterddr @Jackie-Jiang — would appreciate a review when you get a chance! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The 4-argument
splitPartoverload inStringFunctionspreviously allocated a fullString[]viaStringUtils.splitByWholeSeparatoron every call, even when only a single element was needed. This is wasteful in hot query paths wheresplitPartis invoked per-row.Fix
Replace the array-based implementation with index-based forward scanning that extracts only the requested element without materializing all split parts:
Positive index: scan left-to-right, stop at the
limit-th delimiter.Negative index: scan right-to-left using
lastIndexOf, count backward.The behavior and return values are unchanged — this is a pure performance optimization.
Test plan
StringFunctionsTest/splitParttest cases all pass