feat(realtime): support external offsets and shredding-aware reads

[lxy-9602]

Purpose

Linked issue: #158

This change improves realtime writes and reads.

For realtime-enabled writes, external engines must provide a leading non-nullable INT64 _REALTIME_OFFSET:

[_REALTIME_OFFSET, table write fields...]

Offsets must be strictly increasing for each partition-bucket, but gaps are allowed. Paimon persists the exclusive committed offset in the snapshot offset file for WAL recovery. _REALTIME_OFFSET is not written into data files.

Because offsets may be sparse, OffsetRange remains left-closed and right-open but no longer represents the row count.

This PR also moves offset filtering and physical-to-logical conversion into a shared Paimon read pipeline for append-only and primary-key tables. It supports consistent disk and realtime-memory reads for:

• selected-key and shared-shredding MAP access;
• nested projections;
• shredded VARIANT access and binary fallback.

Tests

Added or updated coverage for:

• external and sparse realtime offsets;
• invalid, duplicate, backward, and overflowing offsets;
• commit progress, snapshot offsets, and recovery;
• exact row counting with sparse offsets;
• append-only and primary-key realtime reads;
• MAP and VARIANT projections across disk and memory.

API and Format

Realtime write input and RealtimeStore APIs are updated to support external offsets and shared read processing.
Data-file and snapshot offset-file formats are unchanged.

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[zjw1111]

+1

[SteNicholas]

LGTM.