The ReplicaOfflineMsg grace period is spent on the first message of the process: DSRSShutdownSync latches its timestamp and never resets it

[vharseko]

DSRSShutdownSync.stopInstanceTimestamp records the time of the first ReplicaOfflineMsg a
process ever sends and is never reset, so the grace period it exists for is spent long before
the shutdown it is meant to cover.

The path

opendj-server-legacy/src/main/java/org/opends/server/replication/service/DSRSShutdownSync.java:44-48

public void replicaOfflineMsgSent(DN baseDN)
{
  stopInstanceTimestamp.compareAndSet(0, System.currentTimeMillis());
  replicaOfflineMsgs.add(baseDN);
}

stopInstanceTimestamp is a static AtomicLong, and compareAndSet(0, ...) writes it once per
JVM. canShutdown() at :69-73 is the only reader:

return !replicaOfflineMsgs.contains(baseDN)
    || System.currentTimeMillis() - stopInstanceTimestamp.get() > 5000;

The message is published from ReplicationBroker.stop()
(opendj-server-legacy/src/main/java/org/opends/server/replication/service/ReplicationBroker.java:2684)
→ LDAPReplicationDomain.publishReplicaOfflineMsg(), and broker.stop() is reached by every
disableService(), not only by a shutdown:

• LDAPReplicationDomain.disable(), which processImportBegin() and processRestoreBegin() call -
  so any online import, any restore and any total update trips it;
• ReplicationDomain.restartService() and readAssuredConfig(), so a configuration change does too.

What it costs

Once the timestamp is latched - the first import after startup is enough - now - stopInstanceTimestamp
is far past 5000 ms for the rest of the process lifetime, so canShutdown() returns true at once
for every domain. At a later, real shutdown a collocated replication server's ServerReader /
ServerWriter stops without waiting for the ReplicaOfflineMsg to be forwarded to the other
replication servers, which is exactly what OPENDJ-1453 added this class to prevent: the rest of the
topology is not told that this replica went offline and keeps waiting for changes from it in its
eligibility computations.

Not obvious to fix

The naive fix - set(System.currentTimeMillis()) on every message - swaps the failure over: a domain
which stops and starts its session repeatedly (a fractional configuration change, or the replay
failure recovery of #889) would keep pushing the window forward, and canShutdown() could stay
false for a domain whose message was never forwarded. A per-baseDN timestamp, next to the
replicaOfflineMsgs set which is already per-baseDN, says what is meant: how long ago this domain
announced itself offline.

Found while reviewing #892, which restarts the session of a domain whose replay failed and therefore
reaches replicaOfflineMsgSent() more often than an administrator ever would. That PR leaves this
path as it is.