[core] Add manifest sidecars for partition, row-id and bucket pruning

docs/docs/concepts/spec/manifest.md

@@ -63,6 +63,162 @@ skip manifests before opening them.
 Each extra file belongs exclusively to one manifest. It is retained and cleaned up together with
 that manifest during snapshot, tag, or changelog deletion.
 
+### Manifest Sidecar
+
+With `manifest.sidecar.write` enabled, a manifest writer can create a binary
+`<manifest-file-name>.avro.sidecar` sidecar. Its name is stored in the manifest-list
+record's `_EXTRA_FILES`; the existing Avro schemas and `_VERSION` identifiers are unchanged.
+Readers identify the sidecar by the `.avro.sidecar` suffix among these explicit
+references, not by probing for a derived file name. Other extra-file references are preserved.
+
+With `manifest.sidecar.read` enabled and a partition, row-ID or bucket filter available, readers can use
+the sidecar to select complete Avro blocks before reading manifest entries. Each option
+inherits `manifest-sort.enabled` when unset; an explicit value overrides it independently.
+Since manifest sorting defaults to `false`, sidecar reads and writes are also disabled when
+none of these options is set. Old manifests, null or empty extra-file lists, and lists containing only
+other extra-file types use the normal manifest read path. Missing sidecars and explicit container
+validation failures, such as unsupported versions, checksum mismatches and byte-budget violations,
+also fall back to that path. Each block's partition, row-ID and bucket
+coverage is independently usable; an unavailable dimension cannot exclude a block.
+Java falls back only on `IOException`. If the current thread is interrupted, it instead throws
+`UncheckedIOException` with the original I/O failure. Other exceptions and errors propagate unchanged;
+Java does not inspect causes or suppressed exceptions. PyPaimon explicitly propagates cancellation
+and interruption exception types, including wrapped causes.
+
+Writers choose partition, row-ID and bucket payloads through internal sidecar settings derived from
+the table. Partition coverage is enabled when the supplied partition count is greater than zero;
+otherwise the partition dictionary and partition payloads are omitted. Row-ID coverage follows
+`data-evolution.enabled`. Bucket coverage is enabled when `bucket` is not `-1`, independently
+of data evolution. A disabled payload uses encoding 0, with no length or payload bytes.
+Each payload remains independently usable by readers. These settings do not add
+table options or prevent readers from using payloads already present in existing sidecars.
+
+Version 1 uses the following layout. Container integers and payload integers
+are fixed-width big endian. Encoding IDs are unsigned bytes with separate namespaces.
+
+```text
+magic : 8 bytes                         // ASCII PAIMSCAR
+formatVersion : int                    // 1
+manifestNameHash : 32 bytes             // SHA-256 of the UTF-8 basename
+manifestLength : long
+manifestEntryCount : long               // ADD + DELETE
+avroHeaderLength : int
+avroHeader : bytes                      // original schema, codec and sync marker
+partitionCount : int
+partitionDictionary[]
+  partitionByteLength : int
+  partitionBytes : bytes                // existing manifest BinaryRow serialization
+blockCount : int
+blocks[]                               // original physical order
+  offset : long
+  length : long                         // complete encoded block, including sync marker
+  recordCount : long
+  partitionEncoding : byte
+  if partitionEncoding != 0:
+    partitionPayloadLength : int
+    partitionPayload : bytes
+  rowIdEncoding : byte
+  if rowIdEncoding != 0:
+    rowIdPayloadLength : int
+    rowIdPayload : bytes
+  bucketEncoding : byte
+  if bucketEncoding != 0:
+    bucketPayloadLength : int
+    bucketPayload : bytes
+checksum : 32 bytes                     // SHA-256 of all preceding bytes
+```
+
+The block ID is its position. Its first entry ordinal is the sum of preceding record
+counts and is not stored. Each complete partition tuple appears once in the dictionary,
+including all its fields and nulls. The scan's partition type interprets the existing
+serialized tuple. Partition predicates are evaluated once per dictionary entry.
+
+| Dimension | Encoding | Payload |
+| --- | --- | --- |
+| Any | `0` | Unavailable; only the encoding byte is present. |
+| Partition | `1` | Positive `partitionIdCount: int` followed by sorted unique dictionary IDs (`int`). |
+| Row ID | `1` | Positive `rangeCount: int` followed by sorted disjoint inclusive `(start: long, end: long)` pairs. Coverage may conservatively include gaps. |
+| Bucket | `1` | Positive `pairCount: int` followed by sorted unique `(bucket: int, totalBuckets: int)` pairs. |
+| Any | Other nonzero ID | Skip exactly the bounded payload length; treat only this dimension as unavailable. |
+
+Only nonzero encodings are followed by a length and payload. Payload lengths exclude
+the encoding and length fields. Invalid lengths, known-payload
+framing, dictionary references, interval order, checksums or physical coverage invalidate
+the container. Byte spans must cover the entire original manifest after its header;
+record counts must sum to the manifest entry count. Readers validate the checksum,
+payload framing (including known count/length consistency), and the complete block directory
+even when a block is rejected. Block payload contents are decoded and validated only for
+dimensions still needed by the filters.
+
+Bucket encoding 1 contains a positive `pairCount: int` followed by that many
+`(bucket: int, totalBuckets: int)` pairs. Pairs are sorted by bucket, then totalBuckets,
+and deduplicated. They preserve bucket-count changes between writes; the bucket number
+alone is not sufficient for point lookup after rescaling. A valid pair satisfies
+`0 <= bucket < totalBuckets`. Missing, invalid, negative/synthetic or over-budget bucket
+metadata makes that block's bucket coverage unavailable (encoding 0, no length or payload). Partition
+and row-ID coverage remain independently usable; no mutual-exclusion restriction is imposed.
+
+Readers test bucket-only queries using the existing bucket-selection logic, including
+the total-bucket count. Java uses conservative partition-independent bounds for
+`ManifestBucketFilter`; arbitrary partition-dependent callbacks remain at the entry
+filter stage. An unavailable bucket payload cannot exclude a block. Malformed payload lengths,
+pair counts, ordering or values invalidate the container rather than excluding a block.
+
+All entries contribute, including ADD, DELETE and every file format/column group.
+Row-ID ranges are never expanded into individual values. If an exact union exceeds its
+available byte budget, it becomes the inclusive `[min,max]` envelope with encoding 1. Processing
+continues through the end of the block to extend those bounds and detect unknown row IDs.
+An unknown or invalid row-ID range makes only that block's row-ID payload unavailable.
+Partition budget exhaustion independently makes that block's partition payload unavailable.
+The dictionary can consequently be incomplete for the manifest: a dictionary miss never
+excludes a block with unavailable partition coverage. Later blocks can still use existing IDs.
+
+`manifest.sidecar.max-bytes` bounds the whole serialized container, including the
+partition dictionary and all three payload types. It accepts memory sizes such as
+`16 mb` and, when unset, defaults to twice the configured `manifest.target-file-size`
+(16 MiB with the default 8 MiB manifest target). An explicit sidecar size overrides this default.
+The effective budget is capped at 2147483646 bytes to fit the in-memory byte-array representation.
+Smaller budgets that cannot fit a sidecar skip this optimization; they do not prevent manifest writes
+or reads. Doubling a very large manifest target saturates at the maximum representable memory size.
+The Avro header and block directory share this byte budget without separate size or count limits.
+Writers discard optional row-ID payloads, bucket payloads, then partition payloads/dictionary if necessary,
+to fit the complete directory. If the directory itself cannot fit, no sidecar is published.
+No emitted sidecar omits block descriptors. These are encoded-size bounds; Avro header parsing
+and sidecar construction also incur object/buffer overhead. Query concurrency multiplies per-reader costs.
+
+For conjunctive filters a block is retained only if each dimension is either unavailable
+or matches. Within each block, matching tests row ID, partition, then bucket coverage.
+It skips absent filters and short-circuits after a dimension rejects a block, skipping
+the contents of later payloads. Matches in different dimensions can come
+from different entries in the block, so entry filtering and deletion merging remain
+necessary. Block min/max is derived from the first/last interval before testing the
+individual intervals.
+
+Readers still consume and validate the whole bounded sidecar. A partition-only query
+therefore reads row-ID payload bytes too; payload lengths save decoding work for unknown
+encodings, not storage I/O. Selected compressed blocks are read by byte range with adjacent
+spans coalesced. Existing immutable manifests are not backfilled by enabling the write option.
+
+Java readers share the existing manifest cache for complete sidecar bytes, keyed by the
+explicit sidecar path and subject to the same memory budget and single-file threshold.
+Only successful reads and selections populate the cache. Each query creates independent
+views and reapplies its filters and byte budget; query-specific selections are not cached.
+
+PyPaimon reuses `CachingFileIO` for sidecar bytes. Enable `local-cache.enabled` on the catalog
+and include `meta` in `local-cache.whitelist` (included by default). Files ending in
+`.avro.sidecar`, including custom names, use the same cache as other metadata. The cache
+stores raw byte blocks by full path and block index, sharing `local-cache.max-size` and
+`local-cache.block-size`. Without `local-cache.dir` it uses memory; setting that option
+enables disk caching. Sidecar validation, filters and the read byte budget are reapplied
+on every query. Local caching is disabled by default.
+
+Java selections covering every block can reuse the full-manifest cache; partial selections
+bypass it. PyPaimon explain scans disable sidecar pruning to preserve complete entry counters.
+
+Selected blocks still pass through entry filtering and ADD/DELETE reconciliation. Snapshot,
+tag, changelog, orphan-file and failed-commit cleanup retain or remove the sidecar through
+its extra-file reference together with the owning manifest.
+
 ## Manifest
 
 Data manifests record **ADD** (`0`) and **DELETE** (`1`) entries. Readers reconcile these entries

docs/generated/core_configuration.html

@@ -1113,6 +1113,24 @@
             <td>Boolean</td>
             <td>Whether to skip automatic manifest merging during commit when write-only is true. This also skips automatic manifest sort rewrite. Explicit manifest compaction is not affected.</td>
         </tr>
+        <tr>
+            <td><h5>manifest.sidecar.max-bytes</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>MemorySize</td>
+            <td>Maximum serialized manifest sidecar size, including header and checksum. Defaults to twice manifest.target-file-size. Optional payloads are dropped before omitting a sidecar whose complete block directory cannot fit.</td>
+        </tr>
+        <tr>
+            <td><h5>manifest.sidecar.read</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>Boolean</td>
+            <td>Read optional manifest sidecars for partition, row-id or bucket filters after coarse pruning. Missing or invalid sidecars fall back to manifest reads. Defaults to manifest-sort.enabled when unset.</td>
+        </tr>
+        <tr>
+            <td><h5>manifest.sidecar.write</h5></td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>Boolean</td>
+            <td>Write sidecars with independent partition, row-id and bucket coverage for newly created manifests. Defaults to manifest-sort.enabled when unset.</td>
+        </tr>
         <tr>
             <td><h5>manifest.target-file-size</h5></td>
             <td style="word-wrap: break-word;">8 mb</td>

paimon-api/src/main/java/org/apache/paimon/CoreOptions.java

@@ -510,6 +510,27 @@ public InlineElement getDescription() {
                                     + "in the previous file. This must not exceed "
                                     + "'variant.shredding.minFieldCardinalityRatio'.");
 
+    public static final ConfigOption<Boolean> MANIFEST_SIDECAR_WRITE =
+            key("manifest.sidecar.write")
+                    .booleanType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Write sidecars with independent partition, row-id and bucket coverage for newly created manifests. Defaults to manifest-sort.enabled when unset.");
+
+    public static final ConfigOption<Boolean> MANIFEST_SIDECAR_READ =
+            key("manifest.sidecar.read")
+                    .booleanType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Read optional manifest sidecars for partition, row-id or bucket filters after coarse pruning. Missing or invalid sidecars fall back to manifest reads. Defaults to manifest-sort.enabled when unset.");
+
+    public static final ConfigOption<MemorySize> MANIFEST_SIDECAR_MAX_BYTES =
+            key("manifest.sidecar.max-bytes")
+                    .memoryType()
+                    .noDefaultValue()
+                    .withDescription(
+                            "Maximum serialized manifest sidecar size, including header and checksum. Defaults to twice manifest.target-file-size. Optional payloads are dropped before omitting a sidecar whose complete block directory cannot fit.");
+
     public static final ConfigOption<String> MANIFEST_COMPRESSION =
             key("manifest.compression")
                     .stringType()
@@ -3217,6 +3238,24 @@ public MemorySize manifestTargetSize() {
         return options.get(MANIFEST_TARGET_FILE_SIZE);
     }
 
+    public boolean manifestSidecarWriteEnabled() {
+        return options.getOptional(MANIFEST_SIDECAR_WRITE).orElseGet(this::manifestSortEnabled);
+    }
+
+    public boolean manifestSidecarReadEnabled() {
+        return options.getOptional(MANIFEST_SIDECAR_READ).orElseGet(this::manifestSortEnabled);
+    }
+
+    public MemorySize manifestSidecarMaxSize() {
+        return options.getOptional(MANIFEST_SIDECAR_MAX_BYTES)
+                .orElseGet(
+                        () -> {
+                            long target = manifestTargetSize().getBytes();
+                            return new MemorySize(
+                                    target > Long.MAX_VALUE / 2 ? Long.MAX_VALUE : target * 2);
+                        });
+    }
+
     public MemorySize manifestFullCompactionThresholdSize() {
         return options.get(MANIFEST_FULL_COMPACTION_FILE_SIZE);
     }

paimon-benchmark/paimon-micro-benchmarks/src/test/java/org/apache/paimon/benchmark/ManifestFileSorterBenchmark.java

@@ -31,6 +31,7 @@
 import org.apache.paimon.manifest.ManifestEntry;
 import org.apache.paimon.manifest.ManifestFile;
 import org.apache.paimon.manifest.ManifestFileMeta;
+import org.apache.paimon.manifest.ManifestSidecar;
 import org.apache.paimon.operation.ManifestFileMerger;
 import org.apache.paimon.options.Options;
 import org.apache.paimon.schema.FileSystemSchemaManager;
@@ -300,7 +301,9 @@ private ManifestFile createManifestFile() {
                         "zstd",
                         pathFactory,
                         TARGET_MANIFEST_SIZE,
-                        null)
+                        null,
+                        new ManifestSidecar.Settings(
+                                new CoreOptions(new Options()), PARTITION_TYPE.getFieldCount()))
                 .create();
     }
 

paimon-core/src/main/java/org/apache/paimon/AbstractFileStore.java

@@ -34,6 +34,7 @@
 import org.apache.paimon.manifest.IndexManifestFile;
 import org.apache.paimon.manifest.ManifestFile;
 import org.apache.paimon.manifest.ManifestList;
+import org.apache.paimon.manifest.ManifestSidecar;
 import org.apache.paimon.metastore.AddPartitionCommitCallback;
 import org.apache.paimon.metastore.AddPartitionTagCallback;
 import org.apache.paimon.metastore.ChainTableCommitPreCallback;
@@ -211,7 +212,8 @@ public ManifestFile.Factory manifestFileFactory() {
                 options.manifestCompression(),
                 pathFactory(),
                 options.manifestTargetSize().getBytes(),
-                readManifestCache);
+                readManifestCache,
+                new ManifestSidecar.Settings(options, partitionType.getFieldCount()));
     }
 
     @Override

paimon-core/src/main/java/org/apache/paimon/manifest/BucketFilter.java

@@ -78,6 +78,22 @@ public boolean test(BinaryRow partition, int bucket, int totalBucket) {
                 || totalAwareBucketFilter.test(partition, bucket, totalBucket);
     }
 
+    /** Conservatively checks an indexed pair without inventing a partition for custom filters. */
+    public boolean mayContain(int bucket, int totalBuckets) {
+        if (onlyReadRealBuckets && bucket < 0) {
+            return false;
+        }
+        if (specifiedBucket != null && bucket != specifiedBucket) {
+            return false;
+        }
+        if (bucketFilter != null && !bucketFilter.test(bucket)) {
+            return false;
+        }
+        return !(totalAwareBucketFilter instanceof ManifestBucketFilter)
+                || ((ManifestBucketFilter) totalAwareBucketFilter)
+                        .mayContain(bucket, bucket, totalBuckets);
+    }
+
     /** Conservatively tests whether a manifest's bucket metadata can contain a matching entry. */
     public boolean mayContain(ManifestFileMeta manifest) {
         Integer minBucket = manifest.minBucket();

paimon-core/src/main/java/org/apache/paimon/manifest/ManifestAvroReader.java

@@ -70,6 +70,18 @@ public final class ManifestAvroReader implements AutoCloseable {
         }
     }
 
+    public byte[] headerBytes() {
+        return blockReader.headerBytes();
+    }
+
+    public long blockOffset() {
+        return blockReader.blockOffset();
+    }
+
+    public long blockLength() {
+        return blockReader.blockLength();
+    }
+
     /** Returns whether another raw Avro block is available. */
     public boolean hasNext() throws IOException {
         return blockReader.hasNextBlock();