providing the same schema that read from backward compatible parquet fails: incompatible arrow schema, expected struct got List

[rluvaton]

Describe the bug
When reading a file that was created with older parquet writer (parquet-mr specificlly) and passing a schema that got from ArrowReaderMetadata fails with:

ArrowError("incompatible arrow schema, expected struct got List(Field { name: \"col_15\", data_type: Struct([Field { name: \"col_16\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"col_17\", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"col_18\", data_type: Struct([Field { name: \"col_19\", data_type: Int64, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: \"col_20\", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} })")

To Reproduce

I've added the file in:

• Add backward compat nested file parquet-testing#96

1 liner

Run this in datafusion-cli

select * from 'https://github.com/apache/parquet-testing/raw/6d1dae7ac5dfb23fa1ac1fed5b77d3b919fbb5f8/data/backward_compat_nested.parquet';

Only the relevant parts

This is the reproduction when taking from datafusion only the relevant parts that got to that error

Cargo.toml:

[package]
name = "repro"
version = "0.1.0"
edition = "2024"

[dependencies]
arrow = "56.2.0"
parquet = "56.2.0"
bytes = "1.10.1"

main.rs:

use std::sync::Arc;
use bytes::Bytes;
use parquet::arrow::arrow_reader::{ArrowReaderMetadata, ArrowReaderOptions};

fn main() {
    // The file is the file that added here: https://github.com/apache/parquet-testing/pull/96
    let file_path = "/private/tmp/parquet-testing/data/backward_compat_nested.parquet".to_string();
    
    let mut data = Bytes::from(std::fs::read(file_path).unwrap());

    let mut options = ArrowReaderOptions::new();
    let reader_metadata = ArrowReaderMetadata::load(&mut data, options.clone()).unwrap();

    let physical_file_schema = Arc::clone(reader_metadata.schema());

    // Commenting this out will make the code work
    options = options
        .with_schema(Arc::clone(&physical_file_schema));

    ArrowReaderMetadata::try_new(Arc::clone(reader_metadata.metadata()), options)
            .unwrap();
}

Expected behavior
Should not fail

Additional context
this might be a bug in DataFusion rather than parquet reader here due to backward compatibility the schema was updated to the new version:

• https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#backward-compatibility-rules

I've added the file in:

• Add backward compat nested file parquet-testing#96

[rluvaton]

Fixing it

[rluvaton]

While trying to fix it I found that it not only for struct with repeated but it looks like the type is ignored for primitive when promoted to a list, fixing as well

[alamb]

Could this be related to ?

• Column with List(Struct) causes failed to decode level data for struct array (regression in 56) #8404

[alamb]

Could this be related to ?

• Column with List(Struct) causes failed to decode level data for struct array #8404

I tested, and while similar sounding, the PR here does not fix #8404 unfortunately

• Column with List(Struct) causes failed to decode level data for struct array (regression in 56) #8404 (comment)