perf: improve field indexing in JSON StructArrayDecoder (1.7x speed up) #9086
Conversation
…exing in StructArrayDecoder
…ecoders and field index creation
Weijun-H
changed the title
Weijun-H
changed the title
Weijun-H
changed the title
scovich
left a comment
scovich
left a comment
There was a problem hiding this comment.
Not sure I understand the indexing code well enough to say whether that part is correct, but the idea of using an optional index for field name lookups makes a lot of sense to me.
| } | ||
| } | ||
|
|
||
| fn build_field_index(fields: &Fields) -> Option<HashMap<String, usize>> { |
There was a problem hiding this comment.
qq: Do lifetimes coincide so that we could return Option<HashMap<&str, usize>> instead?
There was a problem hiding this comment.
Yes, the lifetimes do coincide. we can use HashMap<&'a str, usize> by taking fields: &'a Fields as a parameter, which avoids the self-referential struct problem. However, this would require threading the lifetime parameter <'a> through the entire decoder system across many files. Since the lookup performance is identical, I don’t think it’s worth the added complexity.
There was a problem hiding this comment.
maybe it would be a good follow on PR
| use std::fmt::Write; | ||
| use std::sync::Arc; | ||
|
|
||
| fn build_schema(field_count: usize) -> Arc<Schema> { |
There was a problem hiding this comment.
can you please add some comments here with an example of what this code does / what patterns of input it creates?
Also, it would help me to reproduce your results if you could make a separate PR with the benchmarks (so I can compare main to the PR)
There was a problem hiding this comment.
separate benchmark here
| } | ||
| } | ||
|
|
||
| fn build_field_index(fields: &Fields) -> Option<HashMap<String, usize>> { |
There was a problem hiding this comment.
maybe it would be a good follow on PR
alamb
changed the title
3 participants
Which issue does this PR close?
Rationale for this change
Optimize JSON struct decoding on wide objects by reducing per-row allocations and repeated field lookups.
What changes are included in this PR?
Reuse a flat child-position buffer in
StructArrayDecoderand add an optional field-name index for object mode.Skip building the field-name index for list mode; add overflow/allocation checks.
Are these changes tested?
Yes
Are there any user-facing changes?
No