perf(encoding): compact chunk index for sparse cached page state

[Ali2Arslan]

Summary

Follow-up to #7651. The sparse structural scheduler cached a Vec<ChunkMeta> (value count, byte size, byte offset) plus a separate cumulative-value-offset array — about 32 bytes per chunk, with most fields derivable.

Sparse value chunks are mini-blocks, so this replaces that state with the compact MiniBlockChunkIndex introduced in #7651:

• byte ranges come from width-adaptive PrefixSums (u32 when the value buffer fits);
• uniform non-last value counts use arithmetic UniformFlat mapping;
• non-uniform counts use a compact Flat value-prefix array;
• nested-only row metadata is boxed, so its large representation does not inflate every flat/sparse index;
• nested trailer flags use a raw bit-packed Box<[u8]>, avoiding an unnecessary Arrow buffer header.

Metadata is validated once and then rescanned to build only the final prefix sums, avoiding two temporary Vec<u64> allocations per page. The cached state also no longer duplicates num_visible_items, and DeepSizeOf now includes the sparse plan's layer-vector storage and nested allocations.

Memory behavior

The tests now use deep_size_of() (the same total inline + heap accounting used by the cache), rather than checking heap children alone:

• one-chunk pages do not regress;
• multi-chunk UniformFlat and Flat pages are smaller than the legacy representation;
• asymptotic metadata falls from ~32 bytes/chunk to ~4 bytes/chunk (uniform) or ~8 bytes/chunk (non-uniform);
• zero-value pages intentionally retain at most a 32-byte fixed index-header overhead. Avoiding that would require boxing every non-empty index, adding an allocation and pointer indirection to the hot path.

Testing

• cargo test -p lance-encoding --lib sparse — 55 passed
• total-memory coverage for empty, single, uniform, non-uniform, enlarged-final, and nested mappings
• end-to-end non-uniform scheduling and cross-chunk decode coverage
• malformed metadata and existing sparse roundtrip/validation tests pass unchanged
• cargo clippy -p lance-encoding --tests --benches -- -D warnings
• cargo fmt --all --check

Made with Cursor

[coderabbitai]

📝 Walkthrough

Walkthrough

Sparse primitive decoding replaces cached chunk metadata and cumulative offsets with MiniBlockChunkIndex. Nested row mappings now use boxed trailer bitsets, while scheduling, value slicing, and compactness tests use the updated index representations.

Changes

Compact chunk index migration

Layer / File(s)	Summary
Refactor nested row mappings rust/lance-encoding/src/encodings/logical/primitive/chunk_index.rs, rust/lance-encoding/src/encodings/logical/primitive.rs	Nested mappings now wrap prefix sums, item counts, and boxed trailer-bit storage; parsing and index operations use the new representation.
Build and cache the sparse value index rust/lance-encoding/src/encodings/logical/primitive/sparse.rs, rust/lance-encoding/src/encodings/logical/primitive.rs	Removes legacy chunk metadata and offsets, validates metadata-derived ranges, constructs MiniBlockChunkIndex, and caches it in sparse structural state.
Use and validate compact indexes rust/lance-encoding/src/encodings/logical/primitive/sparse.rs, rust/lance-encoding/src/encodings/logical/primitive.rs	Updates chunk scheduling and value slicing to use index lookups, and tests row-mapping shapes, boundary cases, and deep-size accounting.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• lance-format/lance#7651: Continues the mini-block compact chunk-index work in nested mappings and sparse decoding.
• lance-format/lance#7756: Touches sparse value-path integration and per-chunk value metadata.
• lance-format/lance#7889: Modifies the sparse structural page read and value-chunk scheduling pipeline.

Suggested labels: enhancement

Suggested reviewers: xuanwo

Sequence Diagram(s)

sequenceDiagram
  participant PageInitialization
  participant build_value_chunk_index
  participant SparseStructuralCacheableState
  participant schedule_ranges
  participant MiniBlockChunkIndex
  participant append_value_range
  PageInitialization->>build_value_chunk_index: metadata bytes
  build_value_chunk_index->>SparseStructuralCacheableState: constructed value_index
  PageInitialization->>schedule_ranges: cached value_index
  schedule_ranges->>MiniBlockChunkIndex: find_chunk for visible range
  schedule_ranges->>append_value_range: selected chunk range
  append_value_range->>MiniBlockChunkIndex: first_row and items_in_chunk

Loading

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title is concise and accurately summarizes the sparse encoding cache compaction change.
Description check	✅ Passed	The description is clearly related to the chunk-index refactor and memory reduction changes in the PR.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

rust/lance-encoding/src/encodings/logical/primitive/sparse.rs (1)

5654-5670: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Consider covering the enlarged-last-chunk case. The test exercises uniform_flat and flat, but not a UniformFlat page whose last chunk holds more values than the non-last chunks (e.g. [(_, 2), (_, 4)]), which is where the arithmetic row→chunk mapping is most fragile. Adding a case that both initializes and resolves a value inside that last chunk would guard the concern raised on the build_value_chunk_index uniform selection.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@rust/lance-encoding/src/encodings/logical/primitive/sparse.rs` around lines
5654 - 5670, Extend value_chunk_index_stays_compact_across_row_mappings to cover
a UniformFlat layout with a larger final chunk, such as value counts 2 then 4.
Initialize this case through init_value_index and resolve an index targeting a
value in the final chunk, verifying the row-to-chunk mapping succeeds and
remains correct.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@rust/lance-encoding/src/encodings/logical/primitive/sparse.rs`:
- Around line 5654-5670: Extend
value_chunk_index_stays_compact_across_row_mappings to cover a UniformFlat
layout with a larger final chunk, such as value counts 2 then 4. Initialize this
case through init_value_index and resolve an index targeting a value in the
final chunk, verifying the row-to-chunk mapping succeeds and remains correct.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: QUIET

Plan: Pro Plus

Run ID: 12680847-b85e-4db3-b13c-561a3cd669cd

📥 Commits

Reviewing files that changed from the base of the PR and between 3a72f8a and 1ca3eb8.

📒 Files selected for processing (2)

• rust/lance-encoding/src/encodings/logical/primitive.rs
• rust/lance-encoding/src/encodings/logical/primitive/sparse.rs

💤 Files with no reviewable changes (1)

• rust/lance-encoding/src/encodings/logical/primitive.rs

[codecov]

Codecov Report

❌ Patch coverage is 95.20000% with 12 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...encoding/src/encodings/logical/primitive/sparse.rs	94.23%	8 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!