Fix race condition in ChannelCoordinator startup: wait for receiver readiness before reading localMember - #1062
Open
ABin-Huang wants to merge 4 commits into
Open
Conversation
…eadiness Add waitForReady() default method to ChannelReceiver interface and CountDownLatch readiness signaling to NioReceiver, based on latest main.
…dinator Call clusterReceiver.waitForReady() after start() and before getLocalMember(), eliminating the race window where the listener thread may not have entered the select loop yet. Throw ChannelException on timeout or interruption. Also add new i18n messages for timeout/interruption scenarios.
Add TestChannelCoordinatorStartupRace with 5 test cases covering: - Default method backward compatibility - CountDownLatch lifecycle contract - Exact call order verification - Timeout path - Interrupt handling
Fix regressions introduced in the v2 rebuild: ChannelReceiver.java must keep MAX_UDP_SIZE (referenced by NioReceiver.listen()) and its original JavaDoc; LocalStrings.properties must keep all existing keys and the license header. Only the waitForReady() default method, DEFAULT_READY_TIMEOUT_MS and the two new i18n messages are added on top of the current main content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ChannelCoordinator.internalStart()callsclusterReceiver.start()and then immediately callsgetChannel().getLocalMember(false)without waiting for the receiver's background thread to enter the listen loop. This creates a race window where:NioReceiverlistener thread may not have executedsetListen(true)yetmembershipServicemay not have initializedlocalMemberThis can cause intermittent cluster communication failures where nodes cannot connect to each other because the advertised host/port is incorrect.
The original code had a
// synchronize, big time FIXMEcomment at this location.Fix
Added a readiness signaling mechanism based on
CountDownLatch:ChannelReceiverinterface: AddedwaitForReady(long timeout, TimeUnit unit)default method that returnstrueimmediately (backward compatible). AddedDEFAULT_READY_TIMEOUT_MSconstant.NioReceiver: Addedvolatile CountDownLatch readyLatch. A fresh latch with count=1 is created instart()before launching the listener thread. The latch is counted down inlisten()aftersetListen(true).waitForReady()is overridden to await the latch. Latch is reset to count=0 instopListening()to support restarts.ChannelCoordinator: AfterclusterReceiver.start(), callswaitForReady(5000ms)before readinggetLocalMember(). ThrowsChannelExceptionon timeout or interruption. Removed the FIXME comment.Testing
Added
TestChannelCoordinatorStartupRacewith 5 test cases:testWaitForReadyDefaultMethodReturnsImmediately- Verifies backward compatibility of the default methodtestNioReceiverReadyLatchContract- Verifies CountDownLatch lifecycle via reflectiontestChannelCoordinatorWaitsForReceiverBeforeLocalMember- Verifies exact call order: start() → waitForReady() → getLocalMember() → setLocalMemberProperties()testChannelCoordinatorThrowsWhenReceiverNotReady- Verifies ChannelException on timeout and that getLocalMember is NOT calledtestChannelCoordinatorHandlesInterruptedException- Verifies interrupt status is restored and ChannelException is thrownFiles Changed
java/org/apache/catalina/tribes/ChannelReceiver.java- Add waitForReady() default methodjava/org/apache/catalina/tribes/transport/nio/NioReceiver.java- Add CountDownLatch readiness signalingjava/org/apache/catalina/tribes/group/ChannelCoordinator.java- Wait for receiver readiness before reading localMemberjava/org/apache/catalina/tribes/group/LocalStrings.properties- Add new i18n messagestest/org/apache/catalina/tribes/group/TestChannelCoordinatorStartupRace.java- Add unit testsNote: This PR supersedes #1061 and #1060. The branch is based on the latest
main(fork synced to719edeaa).