[CELEBORN-2443] Serialize the peer's own mapIdBitMap - #3826
Open
LuciferYang wants to merge 3 commits into
Open
Conversation
…ocation The peer block of toPbPartitionLocation read location.getMapIdBitMap instead of location.getPeer.getMapIdBitMap, so a round-trip through fromPbPartitionLocation replaced the peer's own bitmap with a copy of the primary's, silently discarding the replica's committed mapIds. All live call sites today serialize allocator-fresh empty bitmaps on both sides (content-identical), so no behavioral change is expected on current flows; this fixes the shared serde primitive and adds a guard test so a future producer carrying divergent committed bitmaps cannot silently lose the peer's. Signed-off-by: yangjie01 <yangjie01@baidu.com>
…packed serde paths
Contributor
|
LGTM. |
Contributor
Author
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3826 +/- ##
============================================
+ Coverage 58.74% 58.78% +0.05%
Complexity 231 231
============================================
Files 399 399
Lines 28058 28065 +7
Branches 2740 2745 +5
============================================
+ Hits 16479 16495 +16
+ Misses 10380 10371 -9
Partials 1199 1199 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
toPbPartitionLocationfills the peer sub-message fromlocation.getPeerfield by field. ThemapIdBitmapline was the exception: it read off the rootlocation. This PR points that line at the peer and rewrites the round-trip tests around it.The tests now cover four cases: a peer with its own bitmap, a root with no bitmap and a peer that has one, a peer with no bitmap that must not inherit the root's, and the same invariant on the packed path (
toPbPackedPartitionLocationsPair), which had no bitmap assertion at all. The unpacked tests also check the restored peer'smodeand the peer/root back link, neither of whichPartitionLocation.equalscompares.Why are the changes needed?
After a round trip, a peer's
mapIdBitMapwas whatever its root carried. When the root has no bitmap and the peer does, the peer's bitmap disappears:Utils.roaringBitmapToByteStringturns both null and empty intoByteString.EMPTY, andbyteStringToRoaringBitmapturnsEMPTYback into null.No live path reads the wrong value today. Only
CommitHandler.collectResultattaches amapIdBitMapto aPartitionLocation, andGetReducerFileGroupResponseships those locations throughtoPbPackedPartitionLocationsPair, which already writes each location's own bitmap. The remaining callers of the unpacked serializer (PbPartitionSplit.oldPartition, Revive, ChangeLocation, and the non-packedWorkerResource) run at allocation or push time, when the bitmap is still empty. So I left the correctness box unchecked, and I would not backport this until someone shows a path that reaches it.Does this PR resolve a correctness bug?
Does this PR introduce any user-facing change?
How was this patch tested?
build/mvn test -pl common -Dsuites=org.apache.celeborn.common.util.PbSerDeUtilsTest: 36 tests pass.I checked both directions by mutating the code. Reverting the production line to
location.getMapIdBitMapturns the three unpacked tests red. ChangingtoPackedPartitionLocationto prefer the peer's bitmap over the location's own turns only the new packed test red.