feat(blob): support reading map blob values by XiaoHongbo-Hope · Pull Request #278 · apache/paimon-cpp

XiaoHongbo-Hope · 2026-09-02T15:49:17Z

Purpose

Linked issue: #283

Support reading Java-compatible MAP<K, BLOB> payloads from blob files through the real Table API and data-evolution paths.

Decode BOOL, integer, DATE, STRING, BINARY, and DECIMAL keys.
Load top-level MAP<K, BLOB> schemas from table JSON while continuing to reject BLOBs in other nested positions.
Treat existing Java MAP<K, BLOB> tables as read-only in C++: reject C++ table creation, FileStoreWrite::Create, and append compaction until writer support is implemented.
Resolve MAP<K, BLOB> placeholders across multiple sequence layers in both inline and descriptor modes.
Preserve null and empty maps/values, and reject malformed metadata, duplicate keys, non-canonical large DECIMAL encodings, invalid UTF-8 STRING keys, and null map keys.
Reject paimon.map.selected-keys for Map Blob reads because filtering can remove the internal data-evolution fallback placeholder.

TIME keys and ARRAY<BLOB> are intentionally outside this PR and should be handled separately with their end-to-end type and format support.

Tests

ninja -C build-release paimon-blob-format-test
build-release/release/paimon-blob-format-test --gtest_filter='BlobFileBatchReaderTest.*:BlobAsDescriptor/BlobFileBatchReaderTest.*' (17 tests passed)
Focused NestedProjectionUtilsTest.GetMapSelectedKeysRejectsMapBlob test (passed)
Changed production and test translation units compiled with -Werror -Wall
cpplint, codespell, and git diff --check

API and Format

No public API under include/ and no storage format changes. This adds read compatibility for the existing Paimon Java MAP<X, BLOB> payload version 1. C++ writer support is unchanged; table creation, regular writes, and append compaction are explicitly rejected.

Arrow Map requires non-null keys, so Java Map Blob payloads containing null keys are rejected during decoding.

Documentation

No documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

duanyyyyyyy · 2026-09-03T08:41:10Z

@lxy-9602 pls take time to review

lxy-9602 · 2026-09-04T07:15:00Z


+Result<std::shared_ptr<arrow::Array>> BlobFileBatchReader::BuildMapBlobArray(
+    int32_t rows_to_read) const {
+    const auto& struct_type = static_cast<const arrow::StructType&>(*target_type_);


This function seems too long. Please break it down into smaller, more fine-grained pieces.

Refactored in 00da44a. Payload parsing/validation, key appends, and value appends are now split into ReadMapBlobPayload, AppendMapBlobKeys, and AppendMapBlobValues; BuildMapBlobArray only coordinates rows and builders.

lxy-9602 · 2026-09-04T07:34:41Z

+    ASSERT_OK_AND_ASSIGN(int64_t written,
+                         output_stream->Write(file_bytes.data(), file_bytes.size()));
+    ASSERT_EQ(file_bytes.size(), written);
+    ASSERT_OK(output_stream->Close());


Consider use file_system->WriteFile

Updated all added golden/corruption fixture writes to use FileSystem::WriteFile in 00da44a.

lxy-9602 · 2026-09-04T07:51:36Z

+        for (int32_t entry = 0; entry < entry_count; ++entry) {
+            const auto key_length = static_cast<int32_t>(key_lengths[entry]);
+            std::vector<uint8_t> key_bytes(key_length);
+            PAIMON_RETURN_NOT_OK(ReadBlobContentAt(key_offset, key_length, key_bytes.data()));


I’m not sure how large key_bytes can get. If it can be large, it would be better to allocate it from the pool, for example via Bytes.

Updated in 00da44a. Key buffers now use pool-backed Bytes; inline value buffers use the same allocation path.

lxy-9602 · 2026-09-04T08:15:30Z

Thank you very much for adding support for map<..., blob>. Would it be possible to open an issue to describe this feature? I’d also like to know whether there is a plan to support writes in the future.

zjw1111 · 2026-09-04T09:23:16Z

Thanks for your work on this! Is there any plan to support reading and writing ARRAY<BLOB> in a follow-up?

lszskye · 2026-09-04T09:57:51Z

+        {"TIME(0)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(3)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(9)", arrow::time32(arrow::TimeUnit::MILLI)},
+        {"TIME(3) WITHOUT TIME ZONE", arrow::time32(arrow::TimeUnit::MILLI)},


Please also add ut for "TIME(6) WITHOUT TIME ZONE"...

Please also add ut for "TIME(6) WITHOUT TIME ZONE"...

Removed TIME support in this PR.

XiaoHongbo-Hope · 2026-09-04T10:03:10Z

@lxy-9602 I opened #283 to describe the feature and current compatibility boundaries. This PR remains read-only; there is no concrete Map Blob writer timeline yet, current case ready only is enough.

@zjw1111 There is no concrete ARRAY<BLOB> plan yet.

lxy-9602

LGTM. Let’s wait for @lszskye to confirm as well.

Copilot

🟢 Approval recommended

The implementation and tests appear cohesive and aligned with the stated read-only scope, with only a minor error-message clarity nit identified.

Pull request overview

Adds reader-side compatibility for Java Paimon MAP<K, BLOB> stored in .blob files so Paimon C++ can load schemas and read these columns via the Table API (including data-evolution fallback across sequence layers), while explicitly rejecting writer/compaction paths until writer support exists.

Changes:

Extend BlobFileBatchReader to decode Map Blob payload v1 (supported key types + inline/descriptor values) and emit an internal placeholder sentinel compatible with the existing fallback reader.
Allow top-level MAP<..., BLOB> in schema validation / JSON loading paths, but reject paimon.map.selected-keys for Map Blob and reject writer creation / append compaction for loaded Map Blob tables.
Add focused unit/integration tests plus a Java-generated Parquet fixture exercising multi-layer fallback.

File summaries

File	Description
test/test_data/parquet/map_blob_java.db/map_blob_java/snapshot/snapshot-1	Adds Java snapshot metadata fixture (snapshot 1).
test/test_data/parquet/map_blob_java.db/map_blob_java/snapshot/snapshot-2	Adds Java snapshot metadata fixture (snapshot 2).
test/test_data/parquet/map_blob_java.db/map_blob_java/snapshot/snapshot-3	Adds Java snapshot metadata fixture (snapshot 3).
test/test_data/parquet/map_blob_java.db/map_blob_java/snapshot/LATEST	Marks latest snapshot id for the fixture.
test/test_data/parquet/map_blob_java.db/map_blob_java/snapshot/EARLIEST	Marks earliest snapshot id for the fixture.
test/test_data/parquet/map_blob_java.db/map_blob_java/schema/schema-0	Adds table schema JSON containing multiple `MAP<K, BLOB>` columns.
test/test_data/parquet/map_blob_java.db/map_blob_java/README.md	Documents the Java fixture provenance and what it covers.
test/inte/blob_table_inte_test.cpp	Adds end-to-end Table API coverage validating Map Blob reads for multiple key types and descriptor/inline modes.
src/paimon/format/blob/blob_file_batch_reader.h	Declares Map Blob parsing/building helpers in the blob batch reader.
src/paimon/format/blob/blob_file_batch_reader.cpp	Implements Map Blob payload decoding, key validation, value loading/descriptor generation, and placeholder emission.
src/paimon/format/blob/blob_file_batch_reader_test.cpp	Adds unit tests for Map Blob decoding, fallback across layers, and corruption/validation cases.
src/paimon/core/utils/nested_projection_utils.cpp	Rejects `paimon.map.selected-keys` when applied to `MAP<..., BLOB>`.
src/paimon/core/utils/nested_projection_utils_test.cpp	Adds regression test covering Map Blob + selected-keys rejection.
src/paimon/core/schema/table_schema.cpp	Rejects creating schemas containing top-level Map Blob for writer paths.
src/paimon/core/schema/table_schema_test.cpp	Adds JSON load/roundtrip test for Map Blob plus writer-creation rejection test.
src/paimon/core/schema/schema_validation_test.cpp	Updates schema validation coverage for shared-shredding + Map Blob scenarios.
src/paimon/core/schema/arrow_schema_validator.cpp	Permits direct blob items in a top-level map when blob is allowed at that level.
src/paimon/core/schema/arrow_schema_validator_test.cpp	Adds validation tests for top-level Map Blob allowed and nested Map Blob rejected.
src/paimon/core/operation/file_store_write.cpp	Rejects writer creation for loaded Map Blob tables.
src/paimon/core/operation/file_store_write_test.cpp	Adds test asserting Map Blob writer creation is rejected.
src/paimon/core/append/append_compact_coordinator.cpp	Rejects append compaction for loaded Map Blob tables.
src/paimon/core/append/append_compact_coordinator_test.cpp	Adds test asserting append compaction rejects Map Blob tables.
src/paimon/common/reader/blob_fallback_batch_reader.h	Documents placeholder semantics for scalar vs Map Blob.
src/paimon/common/reader/blob_fallback_batch_reader.cpp	Extends fallback placeholder detection to Map Blob sentinel shape.
src/paimon/common/data/blob_utils.h	Adds helpers to identify `MAP<..., BLOB>` and to reject writer schemas containing it.
src/paimon/common/data/blob_utils.cpp	Implements `IsMapBlobField` and writer-schema rejection for Map Blob.

Review details

Files reviewed: 26/48 changed files
Comments generated: 1
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    std::shared_ptr<arrow::Field> read_field = arrow_schema->field(0);
+    if (!BlobUtils::IsBlobField(read_field) && !BlobUtils::IsMapBlobField(read_field)) {
+        return Status::Invalid(fmt::format("field {} is not BLOB", read_field->ToString()));
+    }


lxy-9602 · 2026-09-06T13:24:59Z

+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+-->


It seems unnecessary to include a license in the README for the test data.

It looks like you added another compatibility test, which is great! However, this README does not seem to describe the contents of the data or how each snapshot was generated. Without that information, it will be hard for others to reuse this test data in the future. Please refer to the other README files and improve it accordingly.

XiaoHongbo-Hope added 3 commits September 2, 2026 08:48

feat(blob): support reading map blob values

c90feb4

fix(blob): support map blob table and fallback reads

ce4b0b6

fix(blob): harden map blob schema validation

eddb8ab

XiaoHongbo-Hope marked this pull request as ready for review September 3, 2026 05:46

XiaoHongbo-Hope added 4 commits September 2, 2026 23:08

fix(blob): reject writing map blob tables

ba31553

fix(blob): guard append compaction for map blob

128d47c

fix(schema): require millisecond time32 values

29f54be

fix(test): satisfy gcc range loop warnings

aa5b204

Merge branch 'main' into codex/read-map-blob

2f512b5

lxy-9602 reviewed Sep 4, 2026

View reviewed changes

Comment thread src/paimon/format/blob/blob_file_batch_reader.cpp

lxy-9602 reviewed Sep 4, 2026

View reviewed changes

XiaoHongbo-Hope mentioned this pull request Sep 4, 2026

Support reading MAP<K, BLOB> blob files #283

Open

fix(blob): address map blob review feedback

00da44a

lszskye reviewed Sep 4, 2026

View reviewed changes

XiaoHongbo-Hope and others added 2 commits September 4, 2026 06:06

fix(blob): satisfy clang-tidy

635b1d1

Merge branch 'main' into codex/read-map-blob

a1b3061

lxy-9602 previously approved these changes Sep 4, 2026

View reviewed changes

test: expand map blob read coverage

d287f03

XiaoHongbo-Hope dismissed lxy-9602’s stale review via d287f03 September 4, 2026 14:07

SteNicholas requested a lite review from Copilot September 6, 2026 02:15

Copilot started reviewing on behalf of SteNicholas September 6, 2026 02:16 View session

Copilot AI reviewed Sep 6, 2026

View reviewed changes

lxy-9602 reviewed Sep 6, 2026

View reviewed changes

Uh oh!

Conversation

XiaoHongbo-Hope commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

duanyyyyyyy commented Sep 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

lxy-9602 Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxy-9602 commented Sep 4, 2026

Uh oh!

zjw1111 commented Sep 4, 2026

Uh oh!

lszskye Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

XiaoHongbo-Hope commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lxy-9602 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟢 Approval recommended

Uh oh!

lxy-9602 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

lxy-9602 Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

XiaoHongbo-Hope commented Sep 2, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope Sep 4, 2026 •

edited

Loading

XiaoHongbo-Hope commented Sep 4, 2026 •

edited

Loading