Skip to content

[SPARK-55999][SS] Enable forceSnapshotUploadOnLag by default #54847

Open
zifeif2 wants to merge 2 commits intoapache:masterfrom
zifeif2:enable-force-snapshot
Open

[SPARK-55999][SS] Enable forceSnapshotUploadOnLag by default #54847
zifeif2 wants to merge 2 commits intoapache:masterfrom
zifeif2:enable-force-snapshot

Conversation

@zifeif2
Copy link
Contributor

@zifeif2 zifeif2 commented Mar 17, 2026

What changes were proposed in this pull request?

  1. Config default change (SQLConf.scala): forceSnapshotUploadOnLag default false → true
  2. 3 SPARK-51358 tests fixed (StateStoreCoordinatorSuite.scala): Explicitly set forceSnapshotUploadOnLag=false so lag detection tests aren't interfered with by the forced remediation
  3. Cleanup (StateStoreCoordinatorSuite.scala): Removed redundant forceSnapshotUploadOnLag -> "true"

Why are the changes needed?

When state store is lagging in uploading snapshot in maintenance thread, turning on this feature allows state store to upload snapshot in query execution thread, which improves query reliability

Does this PR introduce any user-facing change?

No

How was this patch tested?

The feature is tested in #52773

Was this patch authored or co-authored using generative AI tooling?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant