perf(read): support warmup between data files

[lucasfang]

Purpose

Linked issue: close #289

Reading a primary-key table in merge-on-read mode serializes the first read of every data file behind the consumption of the previous one, because PrefetchFileBatchReaderImpl starts its background thread lazily, inside the first NextBatchWithBitmap(). Within a sorted run the next file's first read starts only once the current file is exhausted, and across a section's k runs LoserTree::InitializeIfNeeded blocks on each leaf's first read in turn. On remote storage that is one round trip per file instead of overlapping them.

This PR adds one optional Warmup() verb to the two reader abstractions and calls it where the consumer already knows what it is going to read next, so that reader's first read is issued while the current one is still being consumed. Warming changes when a file is opened, never what it reads: ordering, merging, filtering, deletion vectors and metrics are untouched.

How far it prepares the next file is a public option, because it trades memory for latency and a scan that stops early - a LIMIT, or a predicate selective enough to empty a split - pays for whatever it already warmed. WarmupLevel sits in include/paimon/utils/prefetch_cache_config.h next to the CacheConfig it interacts with, set through ReadContextBuilder::SetWarmupLevel and read back through ReadContext::GetWarmupLevel. Each level takes the next file one step further along the read pipeline, so a higher one hides more latency and commits more memory:

• NONE - the behavior from before warmup existed: no extra memory, no background thread.
• RAW - fetch only the next file's raw, still-compressed bytes, which overlaps the remote fetch without materializing a decoded batch ahead of the read.
• DECODED (default) - also start the background decode loop, so decoded batches are ready before the file is read.

Changes:

• virtual void Warmup() with an empty default body on BatchReader (public) and on the internal KeyValueRecordReader, so no existing implementation has to change. It returns void on purpose: the caller may warm a reader it never ends up reading, and a hint about a file nobody reads must not be able to fail the read in progress.
• PrefetchFileBatchReaderImpl::Warmup() is the only real implementation and interprets the level: NONE returns at once, RAW calls the new WarmCacheOnce(), DECODED calls EnsureBackgroundThread(), into which the lazy thread start moved so the read path and the warmup path share one start. All three return early when the read ranges are not fresh, a state NextBatchWithBitmap() still rejects, so an unprepared read is reported there and not swallowed here. ReadAheadCache::Init() is not idempotent, so WarmCacheOnce() guards on cache_warmed_, which CleanUp() clears for the next read-range generation.
• Seven FileBatchReader wrappers forward the call - FieldMappingReader, CompleteRowTrackingFieldsBatchReader, ApplyDeletionVectorBatchReader, ApplyBitmapIndexBatchReader, ShreddingFileReader, VectorFileBatchReader and DelegatingPrefetchReader - plus KeyValueDataFileRecordReader on the KeyValue side. DelegatingPrefetchReader skips the call when NeedPrefetch() is false, because GetReader() then bypasses the prefetch layer and forwarding would decode the reader the consumer is itself reading.
• ConcatKeyValueRecordReader warms the reader it is consuming plus one file ahead (kWarmupLookahead = 1) on every NextBatch(), and LoserTree::InitializeIfNeeded() calls the new WarmupLeaves() before its advance loop, in consumption order, so a section of k runs has k files in flight at once - one per leaf, not two.
• MergeFileSplitRead now builds a section's runs with one CreateRawFileReaders call over the flattened file list and hands the readers back by position, the merge path and the raw path both going through it; CreateReaderForRun is removed. Because the matching is positional, the existing assert on the two counts becomes an explicit Status::Invalid: a release build would otherwise shift files into the wrong run instead of failing.
• The level travels with the rest of the read settings through DataFileReadOptions::warmup_level, filled by AbstractSplitRead and FormatTableRead from the ReadContext and forwarded by DataFileReaderFactory::Open(), the one place a production read builds the prefetching reader. Create() no longer defaults the parameter, so a dropped link is a compile error instead of a silent fallback to DECODED.
• System tables rebuild a ReadContext for the data table underneath them and copied every prefetch and cache setting but not this one, so a caller's NONE or RAW came back as DECODED on the table that does the actual reading; each chain is now a named CreateDataReadContext(), which is also what makes it testable.

Two known limits, left as follow-ups: warming does not cross a section boundary, because sections sit inside a SortMergeKeyValueRecordReader holding a SortMergeReader with no warmup verb to forward to; and the min-heap engine gets the intra-run warmup but not the cross-run one, since only LoserTree warms its leaves - loser-tree is the default.

One build fix rides along, plus the CI gap that hid it; both can be split into their own PR if preferred. PAIMON_RETURN_NOT_OK_FROM_ARROW and PAIMON_ASSIGN_OR_RAISE_IMPL_FROM_ARROW called ToPaimonStatus by unqualified name, which does not resolve at global scope, where benchmark/parquet_format_benchmark.cpp keeps its helpers, so every expansion site there failed to compile; both now spell ::paimon::ToPaimonStatus. CI never saw it because it builds nothing under benchmark/, the top-level CMakeLists.txt having added it only under PAIMON_BUILD_BENCHMARKS; it is now added under PAIMON_BUILD_TESTS OR PAIMON_BUILD_BENCHMARKS, so a tests-only build registers the two gtest targets already gated there (ctest count 28 to 30). Building them exposed two dormant defects, fixed here too: the benchmark test linked only the Parquet format library, so the CoreOptions it constructs could not resolve manifest.format - default avro - and all 8 cases failed, so it now links the benchmark's own PAIMON_BENCHMARK_STATIC_LINK_LIBS; and its negative codec assertion wrote a real "lz4" file, which Parquet rejects only mid-write through a Thrift conversion whose default branch is DCHECK(false), an abort in the six Debug and sanitizer CI jobs, so it now pins the same fact at name resolution with no write.

Tests

18 new cases: 17 appended to test files that already build in an existing target, plus a new data_file_reader_factory_test.cpp registered in src/paimon/CMakeLists.txt. MockFileBatchReader gained a Warmup() override and a GetWarmupCount() accessor, which is what makes the forwarding cases possible without a new mock per wrapper.

• Forwarding, one case per link, each asserting the inner reader's count goes 0 to 1: ConcatKeyValueRecordReaderTest.TestWarmupLooksOneReaderAhead, one case per forwarding wrapper, and KeyValueDataFileRecordReaderTest.TestWarmupForwardsToInnerReader. The Concat case also pins that nothing is warmed before a read asks for it, and that a warmed Concat read to the end yields the same KeyValues as the unwarmed one.
• Merge: SortMergeReaderTest.TestLoserTreeWarmsAllLeavesBeforeAdvancingAny asserts a three-leaf tree records exactly warm2, warm1, warm0, read2, read1, read0, and that a second InitializeIfNeeded() records nothing.
• Levels: PrefetchFileBatchReaderImplTest.TestWarmupLevelNone, .TestWarmupLevelRaw and .TestWarmupLevelDecoded, each over {parquet, orc} x {read-ahead cache on, off}, assert whether the background decode loop started and then read every row against the unwarmed expectation; two more cover both branches of the NeedPrefetch() guard; .TestWarmupCacheRearmsForNewReadRangeGeneration covers the cache_warmed_ lifecycle across a RefreshReadRanges().
• Option plumbing: ReadContextTest.TestSetWarmupLevel, SystemTableTest.TestNewReadPropagatesWarmupLevel - for $ro, $audit_log and $binlog, as EXPECT_EQ so one broken chain does not hide another - and DataFileReaderFactoryTest.OpenForwardsWarmupLevelToPrefetchReader.

The assertions are shaped by mutation testing: seven deletions that all compile - the WarmupLeaves() call, all seven wrapper forwards, DECODED degraded to RAW, cache_warmed_.store(false), the NeedPrefetch() guard, and SetWarmupLevel from each system-table chain - each left the full core and common suites green before, and each is now killed by the case that pins it. The changed paths also stay covered by the existing ConcatKeyValueRecordReaderTest, SortMergeReaderTest (both engines), MergeFileSplitReadTest and PrefetchFileBatchReaderImplTest.

Validation, local Release build with PAIMON_BUILD_TESTS=ON and PAIMON_BUILD_BENCHMARKS=ON, all passed:

cmake --build build --target unittest -j 96
cd build && ctest -j 96 -L unittest --output-on-failure
./build/release/paimon-core-test
./build/release/paimon-common-test
./build/release/paimon-read-inte-test
./build/release/paimon-write-and-read-inte-test
./build/release/paimon-pk-compaction-inte-test

30 of 30 targets under the unittest label, including the two the CMakeLists.txt change brings into a tests-only build; paimon-core-test 2053; paimon-common-test 1572 ran, 1570 passed and 2 skipped - the cache-disabled parameters of TestWarmupCacheRearmsForNewReadRangeGeneration, and the only skips in any run; paimon-read-inte-test 294; paimon-write-and-read-inte-test 198; paimon-pk-compaction-inte-test 38. Not run locally, left to CI: the sanitizer jobs and ci/scripts/build_paimon.sh. The sanitizers matter more than usual here, because DECODED warmup and the NeedPrefetch() guard both start work on a background thread ahead of the reader's own thread.

API and Format

Three public API changes under include/paimon/, all additive except one constructor. No storage format and no protocol change: nothing written to disk, to a manifest, to a file footer or over the wire is touched, and warming changes when a file is opened, never what is read.

• reader/batch_reader.h: virtual void Warmup() with an empty default body on the exported BatchReader. Source compatible, but not binary compatible - a new virtual changes the vtable layout and shifts the slot of every virtual after it - so everything must be recompiled together.
• utils/prefetch_cache_config.h: new enum class PAIMON_EXPORT WarmupLevel { NONE, RAW, DECODED }. Purely additive and binary compatible; a scoped enum carries no data and no functions.
• read_context.h: SetWarmupLevel() and GetWarmupLevel() are new non-virtual members and the builder is a pimpl, so sizeof(ReadContextBuilder) is unchanged. ReadContext's public constructor gained a trailing WarmupLevel, source-breaking for a caller that constructs it directly - the class documents going through ReadContextBuilder, and nothing here does - and it gained a private member, so sizeof(ReadContext) changes; no virtuals, so layout only.

Everything else is under src/paimon/ and not exported: KeyValueRecordReader::Warmup(), LoserTree::WarmupLeaves(), WarmCacheOnce(), EnsureBackgroundThread(), the MergeFileSplitRead signature changes, DataFileReadOptions::warmup_level, the two CreateDataReadContext() methods, and PrefetchFileBatchReaderImpl::Create()'s new trailing WarmupLevel parameter, which has no default.

Documentation

The three new methods need no .rst edit of their own: docs/source/api/read.rst already pulls paimon::ReadContextBuilder, paimon::ReadContext and paimon::BatchReader in through doxygenclass with :members: :undoc-members:, and api/file_format.rst does the same for paimon::FileBatchReader, so they render automatically with the doc comments this PR adds.

One documentation line is added: .. doxygenenum:: paimon::WarmupLevel in api/read.rst. Without it no .rst references the new public enum, so the @see WarmupLevel in SetWarmupLevel's doc comment is a dangling reference and the meaning of the three levels appears nowhere in the rendered docs even though the option is public. The repository already uses doxygenenum for paimon::FieldType, paimon::ByteOrder and paimon::SeekOrigin. No user-guide page is affected: user_guide/prefetch.rst documents no user-facing option at all.

Generative AI tooling

Generated-by: Qoder

[lxy-9602]

+1