[Questions] Occasionally, after deploying RabbitMQ, the pod status of openstack-rabbitmq-rabbitmq-1 is CrashLoopBackOff. rm /var/lib/rabbitmq/ and then restarting the pod resolves the issue.

[sheshesheshe]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

other (please specify)

Erlang version used

26.0.x

Operating system (distribution) used

k8s pod，rabbitmq: 3.12.14-management

How is RabbitMQ deployed?

Community Docker image

rabbitmq-diagnostics status output

Because the RabbitMQ pod status is CrashLoopBackOff, I am unable to execute the rabbitmq-diagnostics command.

Logs from node 1 (with sensitive values edited out)

++ echo openstack-rabbitmq-rabbitmq-1
++ awk -F - '{print $NF}'
+ POD_INCREMENT=1
+ '[' 1 -eq 0 ']'
+ '[' -d /var/lib/rabbitmq/mnesia ']'
+ '[' 1 -eq 0 ']'
+ exec rabbitmq-server
2025-02-17 02:56:35.105620+00:00 [warning] <0.132.0> Overriding Erlang cookie using the value set in the environment
2025-02-17 02:56:36.501480+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2025-02-17 02:56:36.501637+00:00 [notice] <0.230.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
�[38;5;87m2025-02-17 02:56:36.502360+00:00 [notice] <0.230.0> Logging: configured log handlers are now ACTIVE�[0m
�[38;5;87m2025-02-17 02:56:36.695624+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped�[0m
�[38;5;87m2025-02-17 02:56:36.696043+00:00 [notice] <0.230.0> Feature flags: checking nodes `rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local` and `rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local` compatibility...�[0m
�[38;5;87m2025-02-17 02:56:36.705979+00:00 [notice] <0.230.0> Feature flags: nodes `rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local` and `rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local` are compatible�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0> Mnesia('rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'): ** ERROR ** (core dumped to file: "/MnesiaCore.rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local_1739_760996_805282")�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>  ** FATAL ** Failed to merge schema: {aborted,�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>                                       {combine_error,schema,�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>                                        ['rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local',�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>                                         'rabbit@openstack-rabbitmq-rabbitmq-2.rabbitmqdiscovery.openstack.svc.cluster.local',�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>                                         'rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local',�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0>                                         'rabbit@openstack-rabbitmq-rabbitmq-0.rabbitmqdiscovery.openstack.svc.cluster.local']}}�[0m
�[38;5;160m2025-02-17 02:56:36.805877+00:00 [error] <0.303.0> �[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> ** Generic server mnesia_subscr terminating�[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> ** Last message in was {'EXIT',<0.305.0>,killed}�[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> ** When Server state == {state,<0.305.0>,#Ref<0.1321328277.1535770625.225283>}�[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> ** Reason for termination ==�[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> ** killed�[0m
�[38;5;160m2025-02-17 02:56:46.812848+00:00 [error] <0.307.0> �[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> ** Generic server mnesia_monitor terminating�[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> ** Last message in was {'EXIT',<0.305.0>,killed}�[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> ** When Server state == {state,<0.305.0>,[],[],true,[],undefined,[],[]}�[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> ** Reason for termination ==�[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> ** killed�[0m
�[38;5;160m2025-02-17 02:56:46.813015+00:00 [error] <0.306.0> �[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> ** Generic server mnesia_recover terminating�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> ** Last message in was {'EXIT',<0.305.0>,killed}�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> ** When Server state == {state,<0.305.0>,undefined,undefined,undefined,0,�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0>                                false,true,[]}�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> ** Reason for termination ==�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> ** killed�[0m
�[38;5;160m2025-02-17 02:56:46.813114+00:00 [error] <0.309.0> �[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     initial call: mnesia_subscr:init/1�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     pid: <0.307.0>�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     registered_name: mnesia_subscr�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     exception exit: killed�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>       in function  gen_server:decode_msg/9 (gen_server.erl, line 909)�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     ancestors: [mnesia_kernel_sup,mnesia_sup,<0.301.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     message_queue_len: 0�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     messages: []�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     links: []�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     status: running�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     heap_size: 2586�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>     reductions: 4573�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:46.813241+00:00 [error] <0.307.0> �[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     initial call: mnesia_recover:init/1�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     pid: <0.309.0>�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     registered_name: mnesia_recover�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     exception exit: killed�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>       in function  gen_server:decode_msg/9 (gen_server.erl, line 909)�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     ancestors: [mnesia_kernel_sup,mnesia_sup,<0.301.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     message_queue_len: 0�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     messages: []�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     links: []�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     status: running�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     heap_size: 4185�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>     reductions: 9735�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:46.813394+00:00 [error] <0.309.0> �[0m

�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     initial call: gen_event:init_it/6�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     pid: <0.303.0>�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     registered_name: mnesia_event�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     exception exit: killed�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>       in function  gen_event:terminate_server/4 (gen_event.erl, line 580)�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     ancestors: [mnesia_sup,<0.301.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     message_queue_len: 1�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     messages: [{notify,�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>                       {mnesia_system_event,�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>                           {mnesia_down,�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>                               'rabbit@openstack-rabbitmq-rabbitmq-1.rabbitmqdiscovery.openstack.svc.cluster.local'}}}]�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     links: []�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     status: running�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     heap_size: 4185�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>     reductions: 10994�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:46.805740+00:00 [error] <0.303.0> �[0m
BOOT FAILED
===========
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     initial call: application_master:init/4�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     pid: <0.300.0>�[0m
Exception during startup:

�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     registered_name: []�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     exception exit: {normal,{mnesia_app,start,[normal,[]]}}�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>       in function  application_master:init/4 (application_master.erl, line 142)�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     ancestors: [<0.299.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     message_queue_len: 1�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     messages: [{'EXIT',<0.301.0>,normal}]�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     links: [<0.299.0>,<0.44.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     status: running�[0m
error:{badmatch,{error,{normal,{mnesia_app,start,[normal,[]]}}}}

    rabbit:run_prelaunch_second_phase/0, line 358
    rabbit:start/2, line 860
    application_master:start_it_old/4, line 293
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     heap_size: 233�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>     reductions: 167�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:46.805957+00:00 [error] <0.300.0> �[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     initial call: mnesia_monitor:init/1�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     pid: <0.306.0>�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     registered_name: mnesia_monitor�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     exception exit: killed�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>       in function  gen_server:decode_msg/9 (gen_server.erl, line 909)�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     ancestors: [mnesia_kernel_sup,mnesia_sup,<0.301.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     message_queue_len: 1�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     messages: [{'$gen_call',{<0.310.0>,�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>                                 #Ref<0.1321328277.1535639553.225857>},�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>                                {close_log,latest_log}}]�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     links: [<26022.273.0>,<26023.385.0>,<0.355.0>]�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     status: running�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     heap_size: 4185�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>     reductions: 8539�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:46.814091+00:00 [error] <0.306.0> �[0m
�[38;5;87m2025-02-17 02:56:46.821310+00:00 [notice] <0.44.0> Application mnesia exited with reason: {normal,{mnesia_app,start,[normal,[]]}}�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> �[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> BOOT FAILED�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> ===========�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> Exception during startup:�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> �[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> error:{badmatch,{error,{normal,{mnesia_app,start,[normal,[]]}}}}�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> �[0m

�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0>     rabbit:run_prelaunch_second_phase/0, line 358�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0>     rabbit:start/2, line 860�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0>     application_master:start_it_old/4, line 293�[0m
�[38;5;160m2025-02-17 02:56:46.821872+00:00 [error] <0.230.0> �[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>   crasher:�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     initial call: application_master:init/4�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     pid: <0.229.0>�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     registered_name: []�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     exception exit: {{badmatch,�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>                          {error,{normal,{mnesia_app,start,[normal,[]]}}}},�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>                      {rabbit,start,[normal,[]]}}�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>       in function  application_master:init/4 (application_master.erl, line 142)�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     ancestors: [<0.228.0>]�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     message_queue_len: 1�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     messages: [{'EXIT',<0.230.0>,normal}]�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     links: [<0.228.0>,<0.44.0>]�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     dictionary: []�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     trap_exit: true�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     status: running�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     heap_size: 233�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     stack_size: 28�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>     reductions: 172�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0>   neighbours:�[0m
�[38;5;160m2025-02-17 02:56:47.823628+00:00 [error] <0.229.0> �[0m
�[38;5;87m2025-02-17 02:56:47.824152+00:00 [notice] <0.44.0> Application rabbit exited with reason: {{badmatch,{error,{normal,{mnesia_app,start,[normal,[]]}}}},{rabbit,start,[normal,[]]}}�[0m
{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{badmatch,{error,{normal,{mnesia_app,start,[normal,[]]}}}},{rabbit,start,[normal,[]]}}}"}
Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{badmatch,{error,{normal,{mnesia_app,start,[normal,[]]}}}},{rabbit,start,[normal,[]]}}})

Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

# PASTE LOG HERE, BETWEEN BACKTICKS

rabbitmq.conf

enabled_plugins: |2

[rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_delayed_message_exchange,rabbitmq_tracing].

rabbitmq-env.conf: |
SERVER_ADDITIONAL_ERL_ARGS="+S 1:1"
rabbitmq.conf: |
cluster_formation.k8s.address_type = hostname
cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_k8s
cluster_partition_handling = autoheal
listeners.ssl.1 = 5672
listeners.tcp = none
log.console = true
loopback_users.guest = false
management.ssl.cacertfile = /etc/rabbitmq/certs/ca.crt
management.ssl.certfile = /etc/rabbitmq/certs/tls.crt
management.ssl.keyfile = /etc/rabbitmq/certs/tls.key
management.load_definitions = /var/lib/rabbitmq/definitions.json
management.ssl.port = 15680
queue_master_locator = min-masters
ssl_options.cacertfile = /etc/rabbitmq/certs/ca.crt
ssl_options.certfile = /etc/rabbitmq/certs/tls.crt
ssl_options.fail_if_no_peer_cert = true
ssl_options.keyfile = /etc/rabbitmq/certs/tls.key
ssl_options.verify = verify_peer

Steps to deploy RabbitMQ cluster

chart：https://opendev.org/openstack/openstack-helm-infra/src/branch/master/rabbitmq
helm upgrade openstack-rabbitmq openstack/rabbitmq --install --create-namespace --namespace openstack --version 3.12.14 --set pod.replicas.server=3

Steps to reproduce the behavior in question

I have a 3 node RabbitMQ cluster. All queues are quorum type.
Occasionally, after deploying RabbitMQ, the following command is used:
helm upgrade openstack-rabbitmq openstack/rabbitmq --install --create-namespace --namespace openstack --version 3.12.14 --set pod.replicas.server=3

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

# PASTE advanced.config HERE, BETWEEN BACKTICKS

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# Relevant parts of K8S deployment that demonstrate how RabbitMQ is deployed
# PASTE YAML HERE, BETWEEN BACKTICKS

What problem are you trying to solve?

I have a 3 node RabbitMQ cluster. All queues are quorum type.
Occasionally, after deploying RabbitMQ, the following command is used:
helm upgrade openstack-rabbitmq openstack/rabbitmq --install --create-namespace --namespace openstack --version 3.12.14 --set pod.replicas.server=3

The log file rabbitmq-1.log indicates the issue as described. I want to know the reasons for this issue, whether it has been fixed in newer versions of RabbitMQ, and if so, which commit this issue. Additionally, I would like to know if there are any workarounds to avoid this problem.

[michaelklishin]

@sheshesheshe you were told earlier that RabbitMQ 3.12 is out of community support. It will be out of support even for paying customers in just a few months.

In the few cases where Mnesia failed to merge its schema, there were different factors in play,
from nodes not being removed from the cluster correctly to abuse of forced node booting on Kubernetes.

We highly recommend using our own cluster Operators on K8S, or at least learning from them and the docs. We won't investigate an issue with a specific 3rd party Helm chart.

Upgrading to 4.0.x and Khepri will fundamentally solve this because Mnesia simply won't be used in such a cluster.