rabbitmq 'failed to merge schema: bad cookie'

[guzzijones]

I was getting this error in rabbitmq pods greater than 0.

Failed to merge schema: Bad cookie in table definition rabbit_user_permission rabbitmq

I ended up finding this solution

Step1: Downscale the statefulset it will not delete the PVC.

kubectl scale statefulsets stackstorm-ha-rabbitmq --replicas=1

Step2: Access the RabbitMQ Pod.

kubectl exec -it stackstorm-ha-rabbitmq-0 -- /bin/bash

Step3: Reset the cluster

rabbitmqctl stop_app
rabbitmqctl force_boot

Step4: Rescale the statefulset

  kubectl scale statefulsets stackstorm-ha-rabbitmq --namespace teps-rabbitmq --replicas=3

[cognifloyd]

Is there something we can change in this chart to help prevent/resolve this issue? Should this just be added to documentation somewhere? Or something changed in the values passed to the rabbitmq subchart?

[ericssonanders]

This was a really hard one for me. I did also have some success with rescaling to 1, allow for the pods to terminate than scale up again. I guess the mnesia db:s was in a split-brain context for me. I ended up recreating the cluster. But, obviously this is nothing you would want happening in a production scenario. Using OpenShift.

[guzzijones]

It looks like is the solution. I am not sure if we can override these settings from bitnami, though.

Combining podManagementPolicy: parallel with a simple "single node" readiness probe mentioned in the docs would be a much safer solution that does not abuse a feature created as a last resort fo