RW refs break replication when master changes

[Serpentian]

The problem

Rebalancing was started, after some time the replication got broken with the following error:

How it happened

This happened as follows: we had a master, which processed some RW requests, after that the master was changed (which is pretty casual thing, when rebalancing is in progress due to high load on storages), then a new master started sending bucket, but replica has RW ref on it, this breaks replication and requires manual intervention from the user (replication becomes stopped and must be manually restarted)

What to do

1. Maybe we can drop rw refs as soon as master becomes replica. However, there may be RW request in progress, which created RW ref, then yielded, master changed, we dropped the ref, then the master is changed back, and we have working RW request without ref, which is dangerous

2. We can wait for RW refs to become 0 before sending the buckets to the new replicaset. Smth similar to the implementation of the map_callro. The problem with that solution is that in map_callro we need to wait after making the bucket SENDING, here we need to wait before any actions. So, there will be sync for RW refs before making bucket SENDING and sync for RO refs after.

3. That's the problem we created ourselves by introducing the checks of the bucket states and we can also fix it by relaxing these checks

---

@Gerold103, @mrForza

[Gerold103]

That's the problem we created ourselves by introducing the checks of the bucket states and we can also fix it by relaxing these checks

The checks can't be relaxed. Certainly not fully. Any solution to drop the refs will only make it worse. They were added to prevent a much worse problem. If you just drop the checks/refs, then a bucket might be sent by another node, become GARBAGE, and would start being deleted in chunks. If the old RW request is still running, it would then see inconsistent data. Incomplete parts of one bucket.

It also might happen that the instance is not read-only (master = true doesn't require that). Then such RW request might even be able to make changes in a SENDING bucket. For the same reason it won't be safe to just allow a SENDING bucket to have RW refs.

The checks reveal the problem, but their removal wouldn't fix it. It would just make the problem invisible until some request returns seemingly impossible trash or breaks the replication in some other way.

The only right solution coming to my mind right now is that the master has to make sure the replicas don't have RW refs. It would have to go to them and check for that explicitly, before doing the sending. For individual buckets right inside bucket_send() and bucket_recv() or in batches somewhere in rebalancer_worker_f(), for example. The checking also then needs to cover storage refs.

Another option is that when a node becomes a master, it has to go to all the other nodes in the replicaset and check

• If they have any RW refs on any bucket or any storage refs.
• If they are all not masters.

Until then the master can't send any buckets at all. Once it sees that the replicas no longer have any RW refs nor storage refs and don't think of themselves as masters, the current master would allow to call bucket_send/recv(). The benefit is that we won't have to check individual buckets. Might be easier to implement.

After the conditions are satisfied, the current master would do a full replication sync (to receive everything up to now from the old masters), and then can send/recv shit.

We would still have the potential issue of one bucket have RW refs on 2 nodes, but it should become not possible, that such a node would receive SENDING status for that bucket from another node. Because the other node would have to wait until the first node has no RW/storage refs at all and is not a master anymore (so it won't make it SENDING either).

Would that work?

[Serpentian]

After the conditions are satisfied, the current master would do a full replication sync (to receive everything up to now from the old masters), and then can send/recv shit.

The problem here is that we cannot even process RW requests until the full replication sync. See #576. The other problem is that master switch usually happens because the old master dies. Then how are we supposed to sync, when the old one is unavailable

Another option is that when a node becomes a master, it has to go to all the other nodes in the replicaset and check ...

What we gonna do, if node is unavailable? We cannot be sure, that there're no RW refs now

[Gerold103]

The other problem is that master switch usually happens because the old master dies. Then how are we supposed to sync, when the old one is unavailable

It should be ok to stop rebalancing if the old master is unavailable. There is no choice, really.

About the issue of the bucket being ACTIVE on the new master in #576 - this is more tricky indeed. Needs more design.