WIP Prototype DataPage extraction API

[alamb]

Which issue does this PR close?

Part of #10806

Rationale for this change

@marvinlanhenke asked some excellent questions on #10806 (comment)

I needed to try out a few things myself so I figured rather than

What changes are included in this PR?

1. Sketch out a StatisticsExtractor API, and a somewhat jenky implementation
2. Update the arrow_statistics tests to have an option to test data page statistics

Are these changes tested?

Yes

Are there any user-facing changes?

[alamb]

This is one way the API could look like

[alamb]

we need to sort out how to make this look better (perhaps by following the lead of the row group iterators that make special iterators for each underlying parquet statistics datatype and then write the relevant code for converting them all to arrow

[marvinlanhenke]

I tried this approach, which works fine. I'd simply refactor whats mentioned in the comment - and extract data type specific iterators (like we already have for the row groups statistics). What do you think @alamb, instead of using array builder?

pub(crate) fn min_page_statistics<'a, I: Iterator<Item = Option<&'a Index>>>(
    data_type: Option<&DataType>,
    iterator: I,
) -> Result<ArrayRef> {
    // Extract this into data type specific iterator e.g. MinInt64PageStatisticsIterator
    let iter = iterator.flat_map(|opt_index| match data_type {
        Some(DataType::Int64) => match opt_index {
            Some(Index::INT64(native_index)) => native_index
                .indexes
                .iter()
                .map(|x| x.min)
                .collect::<Vec<_>>(),
            _ => vec![None],
        },
        // other data_types
        _ => todo!(),
    });

    Ok(Arc::new(Int64Array::from_iter(iter)))

[alamb]

I think an array builder is likely a better approach.

[alamb]

This is kind of a wierd test change, but I was trying to setup a pattern where we didn't have to change all the tests at once

[alamb]

By changing a test to TestBoth it will test both row group indexes and data page indexes

[marvinlanhenke]

I'll guess one reason why we want to pass in the row_group_indexes is due to the iteration over the row_group_indexes from the access_plan here.

We cannot assume we need all indices since access_plan does filter based on should_scan() or not.
Is this correct? If it is, then this was the missing piece in my prototype.

[alamb]

I'll guess one reason why we want to pass in the row_group_indexes is due to the iteration over the row_group_indexes from the access_plan here.

We cannot assume we need all indices since access_plan does filter based on should_scan() or not. Is this correct? If it is, then this was the missing piece in my prototype.

Yes I think that is correct

I am still not super thrilled with this interface (mostly because it is different than row_group_mins etc that takes an interator directly.

However, I couldn't figure out how to make the types workout for making an iterator over Vec ...

[marvinlanhenke]

I had this interface in some version of my prototype, where it would take an iterator over all row_groups directly. However, this cannot be easily integrated with the existing code and the access_plan.row_group_indices. Perhaps, once the StatisticsConverter is fully used in page_filter.rs we can change the interface?

[alamb]

I think this is now inorporated into #10852