Allow a previously reset node to rejoin its original cluster

[SimonUnge]

If a cluster member for whatever reason gets its local state wiped, it has a hard time re-joining the cluster, as the old cluster members will think the node is already a member and reject the request (if mnesia is used).

Proposed Changes

Mnesia: On failure due to 'already a member', ask to leave the cluster first and retry.
Khepri: no-op. Khepri is less strict already, and rabbit_khepri:can_join would accept a join request from a node that is already a member

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

I would like early feedback here, as to if this naive approach is even OK, if there should be a limited set of retries, and if the logic should live in rabbit_mnesia or in rabbit_db_cluster?
It feels a bit wonky that a function called can_join_cluster would also try to leave a cluster and try again, so perhaps it would be better if rabbit_db_cluster:join instead initiates the leave and retry request?

[michaelklishin]

It's a reasonable idea but I think that rabbit_db_cluster:join/2 (or its Mnesia-specific codepath) does seem like a better place. Plus there is rabbit_db_cluster:can_join/1 that might need a comment.

[SimonUnge]

Yeah, I agree the current position is a but odd. I'll move it, with comments!

[michaelklishin]

rabbit_db_cluster now fails gmake dialyze:

rabbit_db_cluster.erl:239:9: The pattern 
          Error = {error, _} can never match since previous clauses completely covered the type 
          {'error', {'inconsistent_cluster', string()}} |
          {'ok', 'already_member' | [atom()]}

[lukebakken]

Works on my machine! Test process:

cd rabbitmq-server
git checkout main
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that cluster is running.

./sbin/rabbitmqctl -n rabbit-2@shostakovich shutdown
rm -rf /tmp/rabbitmq-test-instances/rabbit-2@shostakovich/
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that rabbit-2 is not jointed to the cluster, and that the "thinks that it's a cluster member, but node..." disagrees is in the logs. rabbit-2 starts standalone, eventually.

./sbin/rabbitmqctl -n rabbit-2@shostakovich shutdown
rm -rf /tmp/rabbitmq-test-instances/rabbit-2@shostakovich/
git checkout su_aws/try_to_leave_cluster_before_joining
make FULL=1
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that rabbit-2 has successfully joined the cluster.

[SimonUnge]

Looking into failing tests.

[SimonUnge]

Works on my machine! Test process:

cd rabbitmq-server
git checkout main
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that cluster is running.

./sbin/rabbitmqctl -n rabbit-2@shostakovich shutdown
rm -rf /tmp/rabbitmq-test-instances/rabbit-2@shostakovich/
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that rabbit-2 is not jointed to the cluster, and that the "thinks that it's a cluster member, but node..." disagrees is in the logs. rabbit-2 starts standalone, eventually.

./sbin/rabbitmqctl -n rabbit-2@shostakovich shutdown
rm -rf /tmp/rabbitmq-test-instances/rabbit-2@shostakovich/
git checkout su_aws/try_to_leave_cluster_before_joining
make FULL=1
make PLUGINS='rabbitmq_management' NODES=3 start-cluster

Note that rabbit-2 has successfully joined the cluster.

Just trying to figure out if I messed up some test results that expect this to fail...

[kjnilsson]

Khepri: no-op. Khepri is less strict already, and rabbit_khepri:can_join would accept a join request from a node that is already a member

in this case khepri would first remove itself then join. This ensures it rejoins as a new member.

Anyhow it makes sense that mnesia would also perform a similar set of steps.

[michaelklishin]

The Selenium suite failure is due to an npm dependency installation failure, not anything in this PR.