Skip to content

perf(parquet): reuse page-index bytes through the reader cache - #285

Open
wangyong9999 wants to merge 3 commits into
apache:mainfrom
wangyong9999:perf/parquet-page-index-cache-20260905
Open

perf(parquet): reuse page-index bytes through the reader cache#285
wangyong9999 wants to merge 3 commits into
apache:mainfrom
wangyong9999:perf/parquet-page-index-cache-20260905

Conversation

@wangyong9999

Copy link
Copy Markdown
Contributor

Purpose

Repeated Parquet point reads reuse serialized footers through ReaderBuilder::WithCache, but reload column and offset indexes when each reader is rebuilt. On remote storage this adds index range requests to every lookup.

Reuse immutable page-index bytes through the caller-provided cache. The footer determines eligible ranges; URI, offset and length identify each entry. Page-index entries share the existing DATA_FILE_FOOTER cache budget. Data-page reads and snapshot discovery are unchanged.

Readers, streams and decryptors remain query-local. No process-wide cache, reader pool or new capacity setting is introduced. Missing cache/URI uses the existing uncached path; read and cache errors propagate. Cached footer and page-index allocations retain their memory pool until eviction, since the cache may outlive a reader's custom pool.

No separate issue or design document.

Tests

Added to paimon-parquet-format-test:

  • TestPageIndexBytesSurviveReaderClose: multiple row groups, second reader avoids index storage reads, data reads bypass cache, invalidation reloads indexes, and a temporary pool survives until eviction.
  • TestPointReadReusesFooterAndPageIndexes: two independent point readers return the same row without reloading metadata.
  • TestCachedFooterKeepsAllocatorAliveUntilEviction: footer allocation lifetime with a temporary pool.

Local C++17 syntax validation passed before the allocator-lifetime follow-up. Full compilation and test execution are in progress; CI results will be followed up. No latency improvement is claimed before end-to-end measurement.

API and Format

No public API or storage-format change. Reuses the existing Cache interface and metadata budget.

Documentation

Inline comments explain range eligibility and lifetime. No new user option.

Generative AI tooling

Generated-by: OpenAI Codex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant