feat(parquet): clip nested wrappers during schema pruning - #24315
feat(parquet): clip nested wrappers during schema pruning#24315buraksenn wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24315 +/- ##
==========================================
+ Coverage 81.17% 81.26% +0.09%
==========================================
Files 1109 1112 +3
Lines 388033 391801 +3768
Branches 388033 391801 +3768
==========================================
+ Hits 314992 318407 +3415
- Misses 54501 54682 +181
- Partials 18540 18712 +172 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
kosiew
left a comment
There was a problem hiding this comment.
Thanks for working on this. The new pruning support looks good overall, and I like that RunEndEncoded stays conservative given the positional cast behavior.
I left one non-blocking suggestion around adding a stronger LargeListView reader regression test.
| vec![0_i32, 0, 2].into(), | ||
| vec![2_i32, 0, 1].into(), | ||
| Arc::new(values), | ||
| Some(NullBuffer::from(vec![true, false, true])), |
There was a problem hiding this comment.
Could we also add a reader test for LargeListView similar to the ListView one here? It would be useful to assert the decoded parent and item validity, along with the values. The LargeListView path is a little different because of the 64-bit offsets, so having a reader regression test would give us better coverage than the current type and mask checks alone.
There was a problem hiding this comment.
Thanks for the review, I've added largelistview case as well
Which issue does this PR close?
ListView/Dictionary/FixedSizeListwrappers in nested schema pruning #24122.Rationale for this change
Parquet nested schema pruning currently descends through
Struct,List, andLargeList. Narrowing casts throughListView,LargeListView,FixedSizeList, orDictionarytherefore read physical leaves that the cast never consumes.The issue also listed
RunEndEncoded, but its casts do not use DataFusion's name-based nested cast path. Arrow may map nested struct fields positionally, so clipping those values could change results; they must remain a full read.What changes are included in this PR?
Extend nested schema pruning through matching
ListView,LargeListView,FixedSizeList, andDictionarywrappers while preserving safe full-read fallbacks.Are these changes tested?
Yes, with new and existing tests.
Are there any user-facing changes?
No.