Repository navigation
Conversation
Add a SQL entry point for watermark-based tags, including snapshots retained by existing tags and optional retention. Preserve existing tags on errors. Cover candidate selection and validation through SQL integration tests, and document watermark semantics. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
JingsongLi
left a comment
JingsongLi
left a comment
There was a problem hiding this comment.
Requirement fit: SUPPORTED. Implementation: FINDINGS at 5175997. Pinning a readable version by ingestion watermark has end-to-end value, and the live/tag selection matches the Java reference for nondecreasing watermarks.
Validation: all 58 SQL procedure tests, 18 snapshot-manager tests, and 16 tag-manager tests passed. Rust 1.98.0 formatting passed, and integration with main 7dd674b is conflict-free. An additional independent SQL/HTTP probe used actual SQL-written Parquet and demonstrated the REST-catalog compatibility gap below: SELECT reads the catalog-only latest snapshot, while this new procedure reports that no qualifying snapshot exists. No tag was written by the failing call.
| "Tag '{tag_name}' already exists" | ||
| ))); | ||
| } | ||
| let snapshot = later_or_equal_watermark_from_all(&sm, &tm, watermark) |
There was a problem hiding this comment.
[P2] Resolve watermark candidates from authoritative REST metadata
This new entry point accepts REST tables, but managers(&table) creates a filesystem-only SnapshotManager, and later_or_equal_watermark enumerates only physical snapshot files. REST snapshots can legitimately exist only in the catalog: RESTSnapshotCommit::commit sends the snapshot over HTTP without publishing a snapshot file. I reproduced this with two real SQL-written Parquet snapshots (watermarks 1000/2000), retained snapshot-1 on storage, and snapshot-2 served only by the REST catalog. SELECT id FROM paimon.test_db.t1 ORDER BY id returns both rows, but CALL sys.create_tag_from_watermark(table => 'test_db.t1', tag => 'checkpoint', watermark => 2000) fails with No snapshot found with watermark >= 2000; the call makes no /snapshot request. Thus a readable REST table cannot use the advertised checkpoint operation. Please resolve candidates through the catalog-aware metadata path, or explicitly reject/document unsupported REST tables before selection, with this SQL/HTTP regression. Merely switching the helper to table.snapshot_manager() is insufficient because its watermark search still lists storage files.
Reject REST-managed tables before filesystem candidate selection because their authoritative snapshots may exist only in the catalog. Add a SQL/HTTP regression with real Parquet data and a catalog-only latest snapshot, and document this limitation. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
2 participants
Purpose
SQL can create tags by commit time but cannot pin a version by watermark without manually resolving a snapshot ID.
Brief change log
Add
sys.create_tag_from_watermarkwith live and tag-retained snapshot selection, earlier snapshot IDs for equal watermarks, optional retention, and validation before writing.Tests
API and Format
Adds a SQL procedure and reuses the existing tag format.
Documentation
Documents watermark semantics and usage; corrects timestamp selection wording.