feat(vector): support VECTOR as a non-key value column in Parquet-backed primary-key tables. - #275
Conversation
| if (file_format != "parquet") { | ||
| return; | ||
| } | ||
|
|
There was a problem hiding this comment.
It looks like the current PR is intended to support merge functions for all types. Please update the issue description accordingly, and also add tests for listagg and spill.
There was a problem hiding this comment.
Also, could you add a test for a PK table with a nested type containing a vector field, to verify the recursive row/column conversion for vectors?
ChaomingZhangCN
changed the title
b4fa97f to
05cffcc
Compare
| sort_options, &exec_context)); | ||
| // Arrow's StructArray sorting path may inspect value columns outside the sort keys. Restrict | ||
| // the batch to the requested fields so non-sortable values, such as VECTOR, are never compared. | ||
| std::shared_ptr<arrow::RecordBatch> sort_batch = |
There was a problem hiding this comment.
I’d like to better understand what issues inspect value columns can cause here. Does it lead to incorrect sorting, crashes, or explicit errors? Also, is this considered a bug in Arrow, and is there any ongoing fix for it?
|
|
||
| struct PrimaryKeyVectorRow { | ||
| int64_t primary_key; | ||
| std::optional<std::array<float, 3>> embedding; |
There was a problem hiding this comment.
Please try to avoid using this kind of struct unless the entire test uses a fixed schema throughout. Otherwise, please use simple Arrow JSON to initialize the input and expected Arrow arrays, as it is more straightforward and easier to read.
2 participants
Purpose
Linked issue: #197
Support
VECTORas a non-key value column in Parquet-backed primary-key tables.This PR is scoped to PK tables:
VECTORsupport for data-evolution tables.VECTORremains unsupported as a primary, partition, bucket, sequence, or sequence-group ordering field.The implementation adds FixedSizeList handling across PK writes, spill serialization,
merge/compaction, equality checks, and Arrow read-back.
Tests
git diff --checkAPI and Format
No public API or on-disk format change.
Documentation
Updated the VECTOR type documentation to state support for PK-table value columns and
the continued restriction for data-evolution tables.
Generative AI tooling
Generated-by: Codex (GPT-5)