Skip to content

Forward incompatible types introduced when writing Iceberg data #887

@sungwy

Description

@sungwy

Apache Iceberg version

None

Please describe the bug 🐞

Through the introduction of #807 we have introduced large_* types in the parquet files, which cannot be read using an earlier version of PyIceberg:
TypeError: Unsupported type: large_string

Although the parquet types are the same, there must be an encoding detail that instructs pyarrow to read these as large_* types on read.

Therefore, instead of defaulting to large_* types, we should default the types to small types on write.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions