Rebase and retry the IdealState update in place on a version conflict during tier-relocation rebalance

[Jackie-Jiang]

Summary

Follow-up to #19054. That PR reduces the cost of the version-checked IdealState update losing the compare-and-set during a tier-relocation rebalance of a strict realtime table (upsert/dedup); this PR reduces the cost of recovering from a lost compare-and-set.

Each batch of segment moves is applied with a version-checked IdealState update. On a continuously-ingesting table, consuming-segment commits bump the IdealState version between the read and the write, so the update loses the compare-and-set. Previously every lost compare-and-set fell back to the top of the convergence loop — another ExternalView-convergence wait (~hundreds of ms) plus a target recompute — so the rebalance could make very little progress while ingestion continued.

Change

When the rebalance moves only tier segments (the base placements of the partitions are unchanged — the same condition #19054 uses to skip the target recompute), a concurrent write that does not touch the segments this batch moves cannot invalidate the batch. So on a ZkBadVersionException:

1. Re-read the IdealState.
2. If the concurrent change is disjoint from the segments this batch moves, rebase the batch onto the latest IdealState (reapply the batch's moves, preserving the concurrent changes) and retry the compare-and-set in place — no ExternalView wait, no rebalanceTable.
3. On overlap, re-read failure, or exhausting a bounded number of attempts (MAX_IDEAL_STATE_UPDATE_REBASE_ATTEMPTS), fall back to the convergence loop as before.

On a successful rebase, the target assignment is refreshed from the adopted IdealState so the currentAssignment.equals(targetAssignment) convergence check stays well-defined when a rebase pulls in concurrently added (e.g. new consuming) or removed (e.g. retention) segments.

Net effect: disjoint version churn is absorbed in place, so a tier relocation makes progress against a steady stream of consuming-segment commits instead of repeatedly restarting.

Notes

• The rebase is gated on the same "moves only tier segments" check as Skip full target recompute on IdealState version change for strict realtime rebalance that only moves tier segments #19054, so when the rebalance moves base placements it conservatively keeps the existing fall-back behavior.
• Reviewers: the observer stats handling on a successful rebase (onRollback is skipped since the batch did land) is the subtlest part and the best target for a unit test that injects a version bump between the read and the compare-and-set.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 15.09434% with 45 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.42%. Comparing base (706b772) to head (931f4af).

Files with missing lines	Patch %	Lines
...ntroller/helix/core/rebalance/TableRebalancer.java	15.09%	43 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #19069      +/-   ##
============================================
- Coverage     65.43%   65.42%   -0.01%     
  Complexity     1421     1421              
============================================
  Files          3425     3425              
  Lines        216173   216219      +46     
  Branches      34239    34249      +10     
============================================
+ Hits         141449   141458       +9     
- Misses        63340    63376      +36     
- Partials      11384    11385       +1     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (ø)
integration	100.00% <ø> (ø)
integration1	100.00% <ø> (ø)
integration2	0.00% <ø> (ø)
java-25	65.42% <15.09%> (-0.01%)	⬇️
temurin	65.42% <15.09%> (-0.01%)	⬇️
unittests	65.42% <15.09%> (-0.01%)	⬇️
unittests1	56.85% <ø> (-0.01%)	⬇️
unittests2	37.81% <15.09%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.