feat: add SnapshotUpdate interface for snapshot-producing operations#349
Closed
shangxinli wants to merge 4 commits intoapache:mainfrom
Closed
feat: add SnapshotUpdate interface for snapshot-producing operations#349shangxinli wants to merge 4 commits intoapache:mainfrom
shangxinli wants to merge 4 commits intoapache:mainfrom
Conversation
Add SnapshotUpdate<Derived> interface that extends PendingUpdateTyped<Snapshot> to provide common methods for all updates that create a new table snapshot. This follows the Java Iceberg API pattern where SnapshotUpdate<ThisT> provides a fluent API for operations like AppendFiles, DeleteFiles, and OverwriteFiles. Methods implemented: - Set(): Set summary properties on the snapshot - StageOnly(): Stage snapshot without updating table's current snapshot - DeleteWith(): Set custom file deletion callback for tracking or custom retention - ToBranch(): Commit snapshot to a specific branch (for WAP workflows) Method deferred: - ScanManifestsWith(): Deferred until executor/thread pool infrastructure is available in the codebase. Documented in class comments for future addition. Key features: - Uses CRTP pattern for type-safe fluent API and method chaining - Extends PendingUpdateTyped<Snapshot> to inherit Apply() and Commit() - Protected members allow derived classes to access state - All methods return Derived& for method chaining Testing approach: - Tests verify behavior through public API only (Apply/Commit) - Avoids exposing internal state (no getter methods for protected members) - Mock implementation returns Snapshot with configured properties - 11 comprehensive test cases covering all methods and error paths This interface will be extended by concrete snapshot operations like: - AppendFiles: Add new data files to the table - DeleteFiles: Remove data files from the table - OverwriteFiles: Replace data files in the table - RewriteFiles: Compact and optimize data files
shangxinli
force-pushed
the
SnapshotUpdate
branch
from
4e37cbf to
7c5302d
Compare
Add detailed comparison between C++ and Java SnapshotUpdate implementations. Documents all 6 Java methods and their implementation status in C++. Implementation status: - 4/6 methods implemented (Set, DeleteWith, StageOnly, ToBranch) - 2/6 methods deferred: * ScanManifestsWith: requires executor/thread pool infrastructure * ValidateWith: requires SnapshotAncestryValidator infrastructure Both missing methods have default implementations in Java that throw UnsupportedOperationException, so they are not critical for initial usage. The documentation serves as a reference for: - Current implementation completeness - Future work items - Design decisions and trade-offs - Comparison with Java API patterns
Fix two CI failures:
1. cpp-linter: Replace push_back with emplace_back
- Line 116: deleted_files.emplace_back(path) instead of push_back
- Line 156: deleted_files.emplace_back(path) instead of push_back
- emplace_back is more efficient as it constructs in place
2. License check: Add Apache license header to markdown file
- Add proper HTML comment-style license header to
SNAPSHOT_UPDATE_API_COVERAGE.md
- Matches the format used in other markdown files in the project
All tests still pass (11/11).
clang-format prefers single-line lambdas for simple callbacks. Reformatted DeleteWith() lambda calls to match project style.
Member
|
Close this in favor of #408 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add SnapshotUpdate interface that extends PendingUpdateTyped to provide common methods for all updates that create a new table snapshot. This follows the Java Iceberg API pattern where SnapshotUpdate provides a fluent API for operations like AppendFiles, DeleteFiles, and OverwriteFiles.
Key features:
This interface will be extended by concrete snapshot operations like:
Changes: