Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: adhere write stats configuration #3209

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

ion-elgreco
Copy link
Collaborator

@ion-elgreco ion-elgreco commented Feb 11, 2025

Description

@roeap according to the protocol we should only write struct stats when enabled. Currently we were ignoring this config and always writing struct stats. The table config is now adhered when either of two are disabled.

@github-actions github-actions bot added the binding/rust Issues for the Rust crate label Feb 11, 2025
@ion-elgreco ion-elgreco force-pushed the fix/adhere-write-stats branch from 317766f to 53696e2 Compare February 11, 2025 17:00
@ion-elgreco ion-elgreco force-pushed the fix/adhere-write-stats branch from 53696e2 to 5b26be3 Compare February 11, 2025 17:02
Copy link

codecov bot commented Feb 11, 2025

Codecov Report

Attention: Patch coverage is 90.78947% with 28 lines in your changes missing coverage. Please review.

Project coverage is 72.20%. Comparing base (b3efdfc) to head (e866db8).

Files with missing lines Patch % Lines
crates/core/src/protocol/checkpoints.rs 48.38% 15 Missing and 1 partial ⚠️
crates/core/src/kernel/arrow/mod.rs 95.60% 9 Missing and 3 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3209      +/-   ##
==========================================
+ Coverage   72.10%   72.20%   +0.09%     
==========================================
  Files         138      138              
  Lines       45320    45569     +249     
  Branches    45320    45569     +249     
==========================================
+ Hits        32678    32903     +225     
- Misses      10567    10596      +29     
+ Partials     2075     2070       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@roeap
Copy link
Collaborator

roeap commented Feb 11, 2025

@ion-elgreco one question :).

We can certainly merge this fix the way it is implemented now, but ...

delta-kernel-rs contains some nice facilities to transform schemas in a more structured manner. In some cases implementations can be very simple.

https://github.com/delta-io/delta-kernel-rs/blob/eedfd473f334ac282d23847110f282118b2203eb/kernel/src/scan/data_skipping.rs#L79-L90

While a little more work, this could also brings us forward in adopting kernel etc and simplifying our codebase :).

Using this can of course also be deferred to a later PR.

@ion-elgreco
Copy link
Collaborator Author

ion-elgreco commented Feb 11, 2025

@roeap I can look into that later for sure in a separate PR, but let's merge this to solve some current inconsistenies

@ion-elgreco ion-elgreco force-pushed the fix/adhere-write-stats branch from 5b26be3 to e866db8 Compare February 14, 2025 16:32
@ion-elgreco
Copy link
Collaborator Author

@roeap can we merge this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
binding/rust Issues for the Rust crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Adhere to delta.checkpoint.writeStatsAsStruct
2 participants