Skip to content

Support reading MAP<K, BLOB> blob files #283

Description

@XiaoHongbo-Hope

Motivation

Java Paimon stores MAP<K, BLOB> columns in dedicated .blob files. Paimon C++ should be able to load those schemas and read the existing payload format through the Table API, including data-evolution fallback across sequence layers.

Reader scope

  • Decode payload version 1 for BOOL, integer, DATE, STRING, BINARY, and DECIMAL keys.
  • Support both inline BLOB values and BLOB descriptors.
  • Preserve null maps, empty maps, null values, and empty values.
  • Validate payload bounds, duplicate keys, DECIMAL canonical encoding, UTF-8 STRING keys, and non-null map keys required by Arrow.
  • Resolve placeholders across multiple sequence layers.

paimon.map.selected-keys is intentionally unsupported for Map Blob reads for now because filtering the internal fallback placeholder would lose older-layer data. TIME keys and ARRAY<BLOB> are also outside this issue and should be handled separately with their end-to-end type/format support.

Write support

This issue only tracks read compatibility. C++ table creation, regular writes, and append compaction must reject MAP<K, BLOB> until a Java-compatible Map Blob writer is implemented. There is no concrete writer timeline yet; writer support should be proposed and tracked separately.

Implementation: #278

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions