No test for the change a stopped replay thread hands back to the replication server

[vharseko]

A replay thread which is stopped while it holds a change - num-update-replay-threads is reconfigured on a live domain - hands the change back to the replication server rather than leaving it listed as owned by a thread which is gone. That path has no test.

What is untested

opendj-server-legacy/src/main/java/org/opends/server/replication/plugin/LDAPReplicationDomain.java

private void abandonReplay(CSN csn)                     // :3023
{
  remotePendingChanges.replayFailed(csn);
  if (shutdown.get() || disabled) { return; }
  logger.info(NOTE_REPLAY_ABANDONED_CHANGE, csn, getBaseDN());
  sessionRestartRequested.set(true);
  runRequestedSessionRestarts(false);
}

and the branch which reaches it, in replay(), which also has to publish the ack reporting the error before the hand-back restarts the session. Both were added in #892 (issue #889); the ordering between the ack and the hand-back was got wrong once already and found by re-reading rather than by a test.

Why the obvious test does not work

The trigger needs a replay thread to be inside the in-place retry loop at the moment the thread count changes. A short circuit which keeps failing does not give that: the thread spends most of its time in waitBeforeSessionRestart() between deliveries, and when the change is released by the recovery instead of by abandonReplay() the end state is the same - the change is delivered again and applied. The test would pass whether or not the hand-back happened.

StateMachineTest does change num-update-replay-threads on a live server, but asserts nothing about a change in flight.

What is needed

A test plugin which blocks a replayed operation until the test releases it - the way ShortCircuitPlugin short-circuits one - so a replay thread can be parked inside op.run() while the test reconfigures the replay-thread count. With that, the assertions are:

• the change is not recorded in the ServerState while the thread is stopped;
• the SAFE_READ ack for that delivery carries the replay error rather than being a plain ack;
• the change is delivered again and applied exactly once;
• replayed-updates-failed is not bumped and no UnreplayedChange alert is raised - abandoning is not giving up.

The same machinery would let the disable() race be tested as well.