fix: only use stats for required cols

[ion-elgreco]

Description

The description of the main changes of your pull request

Related Issue(s)

• fixes File pruning issue when filtering on columns without stats and presence of struct fields #3173
• fixes Checkpointing makes filtering on columns not in dataSkippingStatsColumns completely erroneous #3201

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 63 lines in your changes missing coverage. Please review.

Project coverage is 72.12%. Comparing base (cf5f38a) to head (4494d7c).
Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/lib.rs	0.00%	63 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3210      +/-   ##
==========================================
- Coverage   72.18%   72.12%   -0.07%     
==========================================
  Files         138      138              
  Lines       45292    45320      +28     
  Branches    45292    45320      +28     
==========================================
- Hits        32694    32686       -8     
- Misses      10535    10563      +28     
- Partials     2063     2071       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[hntd187]

I think my suggestion about bounding stats collection should be bounded, but otherwise LGTM

[hntd187]

So if no columns are set collect stats for every column? This might be expensive and users might not realize this, might be better to bound it to like 32 (like databricks does) and let the num index cols go higher.

[ion-elgreco]

If no columns are set, we use num_index_cols, where the default is indeed 32. But if it's -1 then it's use all column stats

[ion-elgreco]

delta-rs/crates/core/src/table/config.rs

Lines 43 to 49 in cf5f38a

	/// The number of columns for Delta Lake to collect statistics about for data skipping.
	/// A value of -1 means to collect statistics for all columns. Updating this property does
	/// not automatically collect statistics again; instead, it redefines the statistics schema
	/// of the Delta table. Specifically, it changes the behavior of future statistics collection
	/// (such as during appends and optimizations) as well as data skipping (such as ignoring column
	/// statistics beyond this number, even when such statistics exist).
	DataSkippingNumIndexedCols,

[hntd187]

Sorry @ion-elgreco I removed this from the merge queue just to get your opinion about the change I proposed above.