feat: read v3 row lineage metadata columns#822
Conversation
| auto options = | ||
| MakeReaderOptions(*data_file, io_, projected_schema_, task.residual_filter(), | ||
| name_mapping_, properties_); | ||
| auto options = MakeReaderOptions( |
There was a problem hiding this comment.
REST scan-task JSON appears to round-trip only first-row-id for DataFile; FileScanTasksFromJson constructs the FileScanTask without any data sequence number. That means this call can pass data_file->first_row_id but an empty data_file->data_sequence_number, and MakeLastUpdatedSequenceNumberArray will return nulls for _last_updated_sequence_number.
The row-lineage spec says missing/null _last_updated_sequence_number should be replaced with the data file’s data sequence number.
Disclosure: I used an LLM to help review this PR and draft this comment.
There was a problem hiding this comment.
I looked into the Java REST implementation and OpenAPI. This is not C++-specific: RESTFileScanTaskParser serializes the task file through ContentFileParser, and ContentFileParser only round-trips first-row-id, not the manifest-entry data sequence number. The OpenAPI FileScanTask/DataFile schema also has first-row-id but no field for the data sequence number.
So the failing case is REST-planned scan tasks reading row lineage metadata, where _last_updated_sequence_number needs the manifest entry sequence number but that value is not part of the REST contract. I think the correct fix should start in the REST API / Java parser, then C++ can implement the same field. I don’t think C++ should add a non-standard private field here.
There was a problem hiding this comment.
Agreed, the fix is beyond this PR. We can create a follow-up issue to track.
Synthesize _row_id and _last_updated_sequence_number when reading data files that omit row lineage columns, and preserve physical non-null lineage values when present. Carry inherited data sequence numbers on DataFile metadata so scan planning and readers follow the same model as Java.
| Result<std::shared_ptr<::arrow::Array>> ProjectRowLineageArray( | ||
| const std::shared_ptr<::arrow::Array>& physical_array, int32_t field_id, | ||
| const arrow::MetadataColumnContext& metadata_context, ::arrow::MemoryPool* pool) { | ||
| if (!arrow::HasRowLineageValue(field_id, metadata_context)) { |
There was a problem hiding this comment.
It looks physical array is not examined and its non-null value might not be preserved with first_row_id present but data_sequence_number absent. Both Avro generic decoding and direct decoding have similar issues.
3 participants
Synthesize _row_id and _last_updated_sequence_number when reading data files that omit row lineage columns, and preserve physical non-null lineage values when present.
Carry inherited data sequence numbers on DataFile metadata so scan planning and readers follow the same model as Java.