Add ha rabbit

[YuryHrytsuk]

What do these changes do?

TODO:

• document how to put node under maintenance --> readme
• support single node cluster (for local or tiny deployments) --> done via jinja and iterating over node count
• document how to update erlang cookie
• document autoscaling (joining nodes dynamically on demand)
• how to properly add / remove nodes?
• test rabbit node count >= 3 --> test repo config values unit test
• how to apply new settings in rabbitmq.conf on a running cluster --> in readme
  • avoid causing restart of containers because of config sha change?
• add e2e test monitoring health of the cluster

Clients should properly use HA rabbit

• configure default replication factor for quorum queues? --> via rabbitmq.conf
• how to connect to a multi-node cluster
  • for backenders: make sure client can switch to a different (healthy) nodes in case 1 node fails

Related issue/s

• closes HA self-hosted RabbitMQ #1176

Related PR/s

Checklist

• I tested and it works

[YuryHrytsuk]

Cluster Formation

Source https://www.rabbitmq.com/docs/clustering

---

Ways of Forming a Cluster

• Declaratively by listing cluster nodes in config file
• Declaratively using DNS-based discovery
• Declaratively using AWS (EC2) instance discovery
• Declaratively using Kubernetes discovery
• Declaratively using Consul-based discovery
• Declaratively using etcd-based discovery

Node Names (Identifiers)

• must be unique

Cluster Formation Requirements

• every cluster member must be able to resolve hostnames of every other cluster member, its own hostname, as well as machines on which command line tools such as rabbitmqctl might be used

Ports That Must Be Opened for Clustering and Replication

• 4369: epmd, a helper discovery daemon used by RabbitMQ nodes and CLI tools
• 6000 through 6500: used by RabbitMQ Stream replication
• 25672: used for inter-node and CLI tools communication and is allocated from a dynamic range (limited to a single port by default, computed as AMQP port + 20000)
• 35672-35682: used by CLI tools for communication with nodes and is allocated from a dynamic

Nodes in a Cluster

• Nodes are Equal Peers

For two nodes to be able to communicate they must have the same shared secret called the Erlang cookie.

• Erlang cookie generation should be done at cluster deployment stage ⚠️

Node Counts and Quorum:

• Two node clusters are highly recommended against

Clustering and Clients

Messaging Protocols

• In case of a node failure, clients should be able to reconnect to a different node, recover their topology and continue operation
• Most client libraries accept a list of endpoints

Stream Clients

• RabbitMQ Stream protocol clients behave differently from messaging protocols clients

Queue and Stream Leader Replica Placement

• Queue leaders should be reasonably evenly distributed across cluster nodes