[SPARK-59617][SQL] Reuse a per-evaluator output buffer in JsonExpressionEvalUtils (json_tuple, JSON_TABLE) - #58889
Open
david-mollitor-db wants to merge 1 commit into
Conversation
…ionEvalUtils (json_tuple, JSON_TABLE) `JsonExpressionEvalUtils` has several JSON-extraction evaluators. `GetJsonObjectEvaluator` and `MultiGetJsonObjectEvaluator` already reuse a single `@transient private lazy val outputBuffer` (a `ByteArrayOutputStream`), calling `reset()` before each use. The other two allocated a fresh `ByteArrayOutputStream` on every field/element: - `JsonTupleEvaluator.parseRow` -- one per extracted field (unbounded per row for a wide `json_tuple(...)`). - `JsonTableEvaluator.serializeCurrentValue` -- one per call; the hot caller expands a JSON array one element at a time. Extend the existing `outputBuffer` pattern to those two: each gets a reused instance field and `reset()`s it before each use. A no-arg `ByteArrayOutputStream` starts at 32 bytes and grows by doubling, so this removes a fresh allocation (and its GC) per field/element on the json_tuple / JSON_TABLE eval path. Behavior-preserving: each value's bytes are materialized into a `UTF8String` before the buffer is reused, so output is byte-for-byte identical; the evaluators are per-expression instances evaluated one row at a time, so the reused buffer is never shared concurrently or used re-entrantly. Verified with `JsonExpressionsSuite` and `JsonTableSuite`. Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
JsonExpressionEvalUtilshas several JSON-extraction evaluators. Two of them --GetJsonObjectEvaluatorandMultiGetJsonObjectEvaluator-- already reuse a single@transient private lazy val outputBuffer(aByteArrayOutputStream) across rows, callingreset()before each use. The other two allocate a freshByteArrayOutputStreamon everyfield/element:
JsonTupleEvaluator.parseRow-- a new buffer per extracted field (unbounded per row for a widejson_tuple(...)).JsonTableEvaluator.serializeCurrentValue-- a new buffer per call; the hot caller expands aJSON array one element at a time.
This extends the existing
outputBufferreuse pattern to those two evaluators: each gets aninstance
outputBufferfield and callsreset()before writing to / reading from it.Why are the changes needed?
A no-arg
ByteArrayOutputStreamstarts at the JDK default of 32 bytes and grows by doubling;allocating (and then GC-ing) a fresh one per field/element is steady allocation churn on the
json_tuple/ JSON_TABLE eval path. Reusing one buffer per evaluator -- already the pattern forthe two get-json-object evaluators in the same file -- removes it.
Does this PR introduce any user-facing change?
No. Each value's bytes are fully materialized into a
UTF8String(
UTF8String.fromBytes(outputBuffer.toByteArray)) before the buffer is reused, so the output isbyte-for-byte identical. The evaluators are per-expression instances evaluated one row at a time
(
GenerateExecdrains each row's iterator sequentially), so the reused buffer is never sharedconcurrently or used re-entrantly.
How was this patch tested?
Existing
JsonExpressionsSuite(json_tupleand the get-json-object evaluators) andJsonTableSuite(JSON_TABLE row expansion, including multi-element arrays -- the reused-buffer hotpath) pass. This is a behavior-preserving change, so no new tests were added.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.