Feat: introduce ExecutionPlan::partition_statistics API

[xudong963]

Which issue does this PR close?

• Closes Add statistics_by_partition API to ExecutionPlan #15495
• Closes up: Add statistics_by_partition API to ExecutionPlan #15503

Rationale for this change

• This is a follow up that implements the suggestions on: Add statistics_by_partition API to ExecutionPlan #15503

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

[xudong963]

cc @berkaysynnada PTAL, I didn't see any challenges during refactoring, the process is smooth.

The tests are failing due to #15689

[berkaysynnada]

Great work @xudong963, thank you ! I’ll just ask about specifying deprecation notices for the obsolete API before merging.

[xudong963]

Github is down, my recent update is delayed

[alamb]

Thanks @xudong963 and @berkaysynnada

I also think you should add a note to the upgrade guide for this change (basically people will need to implement partition_statistics if they have implemented statistics -- and I think the compiler may not flag the newly introduced method.

Something that might be a better UX for downstream users might be to NOT provide a default implementation of partition_statistics and force them to implement it:

    fn partition_statistics(&self, partition: Option<usize>) -> Result<Statistics>;

But then they would need some way to avoid the boiler plate code 🤔

[xudong963]

I also think you should add a note to the upgrade guide for this change

Added to #15771

and I think the compiler may not flag the newly introduced method.

Because we marked the old statistics method as deprecated, the compiler will mention users and navigate them to the new method.

Something that might be a better UX for downstream users might be to NOT provide a default implementation of partition_statistics and force them to implement it:

Yes, I've experienced this during upgrading DF45, the version introduced a new method for LexOrdering IIRC, and gave it a default implementation; the default isn't suitable for our case, and I didn't notice the new method; it took me a long time to debug. QAQ

But then they would need some way to avoid the boiler plate code 🤔

Also agree, it's a trade-off, given that the compiler will metion as I said above and we'll also metion the change in the upgrade doc, life will be easier.

Or after merging the PR, we can implement the new method in all places and then remove the default implemation.

[alamb]

Let's get this one in -- it has been outstanding for too long. Thanks again @xudong963 and @berkaysynnada