Nodes are running according to the management dashboard but rabbitmqctl commands display errors

[ghost]

I started a cluster of three nodes yesterday. The rabbitmqctl commands were able to display the status of the cluster as well as stopping and starting the nodes.
Today, when I run the command rabbitmqctl cluster_status I got the following error:

Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.

However, in the management dashboard, all the nodes appear to be running and clients are able to send messages to the server.

Still, I run the command rabbitmqctl start_app and it appears to start the node according to the logs but with some errors. The rabbitmqctl cluster_status doesn't recognize the app running in the target node. Also, rabbitmqctl stop_app doesn't stop any of the nodes that appear to be running in the management console.

The relevant log excerpt seems to be: (full log below)

2021-03-11 08:51:46.067 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.068 [info] <0.11297.3> 
 Starting RabbitMQ 3.8.9 on Erlang 23.2.4
 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com
2021-03-11 08:51:46.069 [info] <0.63.0> supervisor: {<0.63.0>,user_sup}, errorContext: child_terminated, reason: eio, offender: [{pid,<0.64.0>},{mod,user_sup}]
2021-03-11 08:51:46.070 [error] <0.63.0> Supervisor {<0.63.0>,user_sup} had child at module user_sup at <0.64.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.070 [error] <0.63.0> ** Generic server <0.63.0> terminating 
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46.070 [error] <0.63.0> CRASH REPORT Process <0.63.0> with 0 neighbours exited with reason: eio in gen_server:handle_common_reply/8 line 805
2021-03-11 08:51:46.070 [error] <0.49.0> Supervisor kernel_sup had child user started with user_sup:start() at <0.63.0> exit with reason eio in context child_terminated

Full log

2021-03-11 08:51:44.660 [info] <0.11250.3> RabbitMQ is asked to start...
2021-03-11 08:51:44.660 [info] <0.44.0> Application rabbit_common started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.136 [debug] <0.11256.3> Adding Defaults
2021-03-11 08:51:45.136 [debug] <0.11256.3> Right Hand Side Substitutions
2021-03-11 08:51:45.136 [debug] <0.11256.3> Applying Datatypes
2021-03-11 08:51:45.137 [debug] <0.11256.3> Validation
2021-03-11 08:51:45.151 [debug] <0.11256.3> Applied 1:1 Mappings
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> "rabbit.writer_gc_threshold" in Translations to drop...
2021-03-11 08:51:45.151 [debug] <0.11256.3> Running translation for rabbit.loopback_users
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.ssl_options.ciphers" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.auth_mechanisms" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.auth_backends" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_vhost" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_user" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_pass" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_permissions" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.default_user_tags" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.connection_max" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.152 [debug] <0.11256.3> "rabbit.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.vm_memory_high_watermark" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.disk_free_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_partition_handling" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> Running translation for rabbit.cluster_formation.peer_discovery_backend
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.node_type" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.randomized_startup_delay_range" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> Running translation for rabbit.cluster_nodes
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_dns.hostname" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "rabbit.queue_master_locator" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "syslog.dest_host" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "syslog.protocol" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.process_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.port_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.gc_ms_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.schedule_ms_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.heap_word_limit" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.busy_port" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "sysmon_handler.busy_dist_port" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.segment_max_entries" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.wal_max_size_bytes" in Translations to drop...
2021-03-11 08:51:45.153 [debug] <0.11256.3> "ra.wal_max_entries" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.wal_hibernate_after" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.wal_max_batch_size" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.snapshot_chunk_size" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "ra.data_dir" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_amqp1_0.default_vhost" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_cache.cached_backend" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.servers" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.dn_lookup_bind" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.other_bind" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_auth_backend_ldap.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_event_exchange.vhost" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.cacerts" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.versions" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.cert" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.dh" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.listener.ssl_opts.key" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.cors_allow_origins" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.cors_max_age" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.content_security_policy" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.strict_transport_security" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_management.sample_retention_policies" in Translations to drop...
2021-03-11 08:51:45.154 [debug] <0.11256.3> "rabbitmq_mqtt.default_user" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.default_pass" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.vhost" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.exchange" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.subscription_ttl" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.rabbit.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbitmq_mqtt.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_ec2_region" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_use_private_ip" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_access_key" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_secret_key" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_autoscaling" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_aws.aws_ec2_tags" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.node_cleanup.cleanup_interval" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.node_cleanup.cleanup_only_log_warning" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.http_proxy" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.https_proxy" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.proxy.proxy_exclusions" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_host" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_port" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_scheme" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_acl_token" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.cluster_name" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_auto" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_nic" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_addr_nodename" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_domain" in Translations to drop...
2021-03-11 08:51:45.155 [debug] <0.11256.3> Running translation for rabbit.cluster_formation.peer_discovery_consul.consul_include_nodes_with_warnings
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_ttl" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_deregister_after" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_use_longname" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_tags" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_svc_meta" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.consul_lock_prefix" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_consul.lock_wait_time" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.endpoints" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_host" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_scheme" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_prefix" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.cluster_name" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_node_ttl" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.lock_wait_time" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_username" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.etcd_password" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_etcd.ssl_options.ciphers" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_host" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_port" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_scheme" in Translations to drop...
2021-03-11 08:51:45.156 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_token_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_cert_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_namespace_path" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_service_name" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_address_type" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbit.cluster_formation.peer_discovery_k8s.k8s_hostname_suffix" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_prometheus.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_prometheus.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listeners" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listen_options" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.tcp_listen_options.linger" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.ssl_listeners" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.default_vhost" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_stomp.default_topic_exchange" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.providers" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.cacerts" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.cert" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.dh" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.key" in Translations to drop...
2021-03-11 08:51:45.157 [debug] <0.11256.3> "rabbitmq_trust_store.ssl_options.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.tcp_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_mqtt.cowboy_opts.idle_timeout" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.tcp_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config.versions" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.ssl_config.ciphers" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> "rabbitmq_web_stomp.cowboy_opts.idle_timeout" in Translations to drop...
2021-03-11 08:51:45.158 [debug] <0.11256.3> Applied Translations
2021-03-11 08:51:45.171 [info] <0.44.0> Application rabbitmq_prelaunch started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.185 [info] <0.44.0> Application os_mon started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.209 [info] <0.11281.3> ra: meta data store initialised. 3 record(s) recovered
2021-03-11 08:51:45.296 [info] <0.44.0> Application ra started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.297 [info] <0.44.0> Application sysmon_handler started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.307 [info] <0.11297.3> Stopping application 'syslog'
2021-03-11 08:51:45.307 [info] <0.11297.3> Stopping application 'lager'
2021-03-11 08:51:45.322 [debug] <0.11312.3> Lager installed handler error_logger_lager_h into error_logger
2021-03-11 08:51:45.325 [debug] <0.11315.3> Lager installed handler lager_forwarder_backend into error_logger_lager_event
2021-03-11 08:51:45.325 [debug] <0.11318.3> Lager installed handler lager_forwarder_backend into rabbit_log_lager_event
2021-03-11 08:51:45.325 [debug] <0.11321.3> Lager installed handler lager_forwarder_backend into rabbit_log_channel_lager_event
2021-03-11 08:51:45.325 [debug] <0.11324.3> Lager installed handler lager_forwarder_backend into rabbit_log_connection_lager_event
2021-03-11 08:51:45.325 [debug] <0.11327.3> Lager installed handler lager_forwarder_backend into rabbit_log_feature_flags_lager_event
2021-03-11 08:51:45.325 [debug] <0.11330.3> Lager installed handler lager_forwarder_backend into rabbit_log_federation_lager_event
2021-03-11 08:51:45.325 [debug] <0.11333.3> Lager installed handler lager_forwarder_backend into rabbit_log_ldap_lager_event
2021-03-11 08:51:45.325 [debug] <0.11336.3> Lager installed handler lager_forwarder_backend into rabbit_log_mirroring_lager_event
2021-03-11 08:51:45.325 [debug] <0.11339.3> Lager installed handler lager_forwarder_backend into rabbit_log_prelaunch_lager_event
2021-03-11 08:51:45.325 [debug] <0.11342.3> Lager installed handler lager_forwarder_backend into rabbit_log_queue_lager_event
2021-03-11 08:51:45.325 [debug] <0.11345.3> Lager installed handler lager_forwarder_backend into rabbit_log_ra_lager_event
2021-03-11 08:51:45.325 [debug] <0.11348.3> Lager installed handler lager_forwarder_backend into rabbit_log_shovel_lager_event
2021-03-11 08:51:45.355 [info] <0.44.0> Application lager started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.360 [info] <0.11297.3> Log file opened with Lager
2021-03-11 08:51:45.363 [debug] <0.11351.3> Lager installed handler {lager_file_backend,
                            "/exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252_upgrade.log"} into rabbit_log_upgrade_lager_event
2021-03-11 08:51:45.412 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:45.423 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-03-11 08:51:45.423 [info] <0.44.0> Application mnesia exited with reason: stopped
2021-03-11 08:51:45.823 [debug] <0.11307.3> Lager installed handler lager_backend_throttle into lager_event
2021-03-11 08:51:46.067 [info] <0.44.0> Application mnesia started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.068 [info] <0.11297.3> 
 Starting RabbitMQ 3.8.9 on Erlang 23.2.4
 Copyright (c) 2007-2020 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com
2021-03-11 08:51:46.069 [info] <0.63.0> supervisor: {<0.63.0>,user_sup}, errorContext: child_terminated, reason: eio, offender: [{pid,<0.64.0>},{mod,user_sup}]
2021-03-11 08:51:46.070 [error] <0.63.0> Supervisor {<0.63.0>,user_sup} had child at module user_sup at <0.64.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.070 [error] <0.63.0> ** Generic server <0.63.0> terminating 
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46.070 [error] <0.63.0> CRASH REPORT Process <0.63.0> with 0 neighbours exited with reason: eio in gen_server:handle_common_reply/8 line 805
2021-03-11 08:51:46.070 [error] <0.49.0> Supervisor kernel_sup had child user started with user_sup:start() at <0.63.0> exit with reason eio in context child_terminated
2021-03-11 08:51:46.072 [info] <0.11297.3> 
 node           : rabbit@cs04r-sc-vserv-252
 home dir       : /exports/rmq/2021-03-10
 config file(s) : /exports/rmq/2021-03-10/rabbitmq.conf
 cookie hash    : YutjBenRobSKMf92dFCNTQ==
 log(s)         : /exports/rmq/2021-03-10/log/[email protected]
                : /exports/rmq/2021-03-10/log/rabbit@cs04r-sc-vserv-252_upgrade.log
 database dir   : /exports/rmq/2021-03-10/mnesia/rabbit@cs04r-sc-vserv-252
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step pre_boot defined by app rabbit
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step rabbit_core_metrics defined by app rabbit
2021-03-11 08:51:46.087 [info] <0.11297.3> Running boot step rabbit_alarm defined by app rabbit
2021-03-11 08:51:46.130 [info] <0.11425.3> Memory high watermark set to 1515 MiB (1589518336 bytes) of 3789 MiB (3973795840 bytes) total
2021-03-11 08:51:46.137 [info] <0.11427.3> Enabling free disk space monitoring
2021-03-11 08:51:46.137 [info] <0.11427.3> Disk free limit set to 50MB
2021-03-11 08:51:46.142 [info] <0.11297.3> Running boot step code_server_cache defined by app rabbit
2021-03-11 08:51:46.142 [info] <0.11297.3> Running boot step file_handle_cache defined by app rabbit
2021-03-11 08:51:46.142 [info] <0.11430.3> Limiting to approx 927 file handles (832 sockets)
2021-03-11 08:51:46.142 [info] <0.11431.3> FHC read buffering:  OFF
2021-03-11 08:51:46.142 [info] <0.11431.3> FHC write buffering: ON
2021-03-11 08:51:46.144 [info] <0.11297.3> Running boot step worker_pool defined by app rabbit
2021-03-11 08:51:46.144 [info] <0.11414.3> Will use 1 processes for default worker pool
2021-03-11 08:51:46.144 [info] <0.11414.3> Starting worker pool 'worker_pool' with 1 processes in it
2021-03-11 08:51:46.145 [info] <0.11297.3> Running boot step database defined by app rabbit
2021-03-11 08:51:46.384 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.480 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.481 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.481 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.489 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.489 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.520 [info] <0.11297.3> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-03-11 08:51:46.520 [info] <0.11297.3> Successfully synced tables from a peer
2021-03-11 08:51:46.520 [info] <0.11297.3> Will register with peer discovery backend rabbit_peer_discovery_classic_config
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step database_sync defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step feature_flags defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step codec_correctness_check defined by app rabbit
2021-03-11 08:51:46.520 [info] <0.11297.3> Running boot step external_infrastructure defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_registry defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_queue_location_random defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_event defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-03-11 08:51:46.521 [info] <0.11297.3> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_priority_queue defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step kernel_ready defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-03-11 08:51:46.522 [info] <0.11297.3> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-03-11 08:51:46.524 [info] <0.11526.3> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-03-11 08:51:46.524 [info] <0.11297.3> Running boot step guid_generator defined by app rabbit
2021-03-11 08:51:46.534 [info] <0.11297.3> Running boot step rabbit_node_monitor defined by app rabbit
2021-03-11 08:51:46.539 [info] <0.11530.3> Starting rabbit_node_monitor
2021-03-11 08:51:46.539 [info] <0.11297.3> Running boot step delegate_sup defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step rabbit_memory_monitor defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step core_initialized defined by app rabbit
2021-03-11 08:51:46.541 [info] <0.11297.3> Running boot step upgrade_queues defined by app rabbit
2021-03-11 08:51:46.564 [info] <0.11297.3> Running boot step rabbit_connection_tracking defined by app rabbit
2021-03-11 08:51:46.564 [info] <0.11297.3> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-03-11 08:51:46.565 [info] <0.11297.3> Running boot step rabbit_policies defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_policy defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step rabbit_vhost_limit defined by app rabbit
2021-03-11 08:51:46.566 [info] <0.11297.3> Running boot step recovery defined by app rabbit
2021-03-11 08:51:46.567 [info] <0.11559.3> Making sure data directory '/exports/rmq/2021-03-10/mnesia/rabbit@cs04r-sc-vserv-252/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-03-11 08:51:46.575 [info] <0.11559.3> Starting message stores for vhost '/'
2021-03-11 08:51:46.575 [info] <0.11563.3> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-03-11 08:51:46.578 [info] <0.11559.3> Started message store of type transient for vhost '/'
2021-03-11 08:51:46.578 [info] <0.11567.3> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-03-11 08:51:46.580 [info] <0.11559.3> Started message store of type persistent for vhost '/'
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step empty_db_check defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Will not seed default virtual host and user: have definitions to load...
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step rabbit_looking_glass defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step background_gc defined by app rabbit
2021-03-11 08:51:46.832 [info] <0.11297.3> Running boot step connection_tracking defined by app rabbit
2021-03-11 08:51:46.834 [info] <0.11297.3> Setting up a table for connection tracking on this node: 'tracked_connection_on_node_rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.844 [info] <0.11297.3> Setting up a table for per-vhost connection counting on this node: 'tracked_connection_per_vhost_on_node_rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step routing_ready defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step pre_flight defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step notify_cluster defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step networking defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step definition_import_worker_pool defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11414.3> Starting worker pool 'definition_import_pool' with 1 processes in it
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step cluster_name defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.11297.3> Running boot step direct_client defined by app rabbit
2021-03-11 08:51:46.845 [info] <0.44.0> Application rabbit started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:46.847 [info] <0.11530.3> rabbit on node 'rabbit@cs04r-sc-vserv-254' up
2021-03-11 08:51:47.175 [notice] <0.11577.3> queue 'transient.destination' in vhost '/': candidate -> leader in term: 2 machine version: 0
2021-03-11 08:51:48.160 [info] <0.11606.3> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
2021-03-11 08:51:48.160 [info] <0.11606.3> Management plugin: using rates mode 'basic'
2021-03-11 08:51:48.221 [info] <0.44.0> Application rabbitmq_management_agent started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.235 [info] <0.44.0> Application cowlib started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.247 [info] <0.44.0> Application cowboy started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.260 [info] <0.44.0> Application rabbitmq_web_dispatch started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.272 [info] <0.44.0> Application amqp_client started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.283 [info] <0.11606.3> Running boot step rabbit_mgmt_reset_handler defined by app rabbitmq_management
2021-03-11 08:51:48.283 [info] <0.11606.3> Running boot step rabbit_management_load_definitions defined by app rabbitmq_management
2021-03-11 08:51:48.309 [info] <0.11674.3> Management plugin: HTTP (non-TLS) listener started on port 15672
2021-03-11 08:51:48.309 [info] <0.11780.3> Statistics database started.
2021-03-11 08:51:48.309 [info] <0.11779.3> Starting worker pool 'management_worker_pool' with 3 processes in it
2021-03-11 08:51:48.310 [info] <0.44.0> Application rabbitmq_management started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.341 [info] <0.44.0> Application prometheus started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:51:48.722 [notice] <0.11582.3> queue 'queue.status' in vhost '/': candidate -> leader in term: 26 machine version: 0
2021-03-11 08:52:40.848 [info] <0.11530.3> rabbit on node 'rabbit@cs04r-sc-vserv-253' up
2021-03-11 08:53:28.777 [info] <0.12374.3> MQTT retained message store: rabbit_mqtt_retained_msg_store_dets
2021-03-11 08:53:28.779 [info] <0.12392.3> started MQTT TCP listener on [::]:1883
2021-03-11 08:53:28.781 [info] <0.44.0> Application rabbitmq_mqtt started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.799 [info] <0.12395.3> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
2021-03-11 08:53:28.799 [info] <0.44.0> Application rabbitmq_prometheus started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.810 [info] <0.12502.3> rabbit_stomp: default user 'guest' enabled
2021-03-11 08:53:28.812 [info] <0.12518.3> started STOMP TCP listener on [::]:61613
2021-03-11 08:53:28.813 [info] <0.11606.3> Ready to start client connection listeners
2021-03-11 08:53:28.813 [info] <0.44.0> Application rabbitmq_stomp started on node 'rabbit@cs04r-sc-vserv-252'
2021-03-11 08:53:28.817 [info] <0.12533.3> started TCP listener on [::]:5672
2021-03-11 08:53:28.819 [notice] <0.11304.3> Changed loghwm of /exports/rmq/2021-03-10/log/[email protected] to 500000000000000000000000000000000000000000000000
2021-03-11 08:53:29.139 [notice] <0.12359.3> mqtt_node: candidate -> leader in term: 29 machine version: 0
2021-03-11 08:53:29.236 [info] <0.11606.3> Server startup complete; 6 plugins started.
 * rabbitmq_stomp
 * rabbitmq_prometheus
 * rabbitmq_mqtt
 * rabbitmq_management
 * rabbitmq_web_dispatch
 * rabbitmq_management_agent

And this is from the crash.log

2021-03-11 08:51:46 =SUPERVISOR REPORT====
     Supervisor: {<0.63.0>,user_sup}
     Context:    child_terminated
     Reason:     eio
     Offender:   [{pid,<0.64.0>},{mod,user_sup}]

2021-03-11 08:51:46 =ERROR REPORT====
** Generic server <0.63.0> terminating 
** Last message in was {'EXIT',<0.64.0>,eio}
** When Server state == {state,user_sup,undefined,<0.64.0>,{<0.63.0>,user_sup}}
** Reason for termination ==
** eio
2021-03-11 08:51:46 =CRASH REPORT====
  crasher:
    initial call: supervisor_bridge:user_sup/1
    pid: <0.63.0>
    registered_name: []
    exception exit: {eio,[{gen_server,handle_common_reply,8,[{file,"gen_server.erl"},{line,805}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,226}]}]}
    ancestors: [kernel_sup,<0.47.0>]
    message_queue_len: 0
    messages: []
    links: [<0.49.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 1598
    stack_size: 28
    reductions: 1181
  neighbours:
2021-03-11 08:51:46 =SUPERVISOR REPORT====
     Supervisor: {local,kernel_sup}
     Context:    child_terminated
     Reason:     eio
     Offender:   [{pid,<0.63.0>},{id,user},{mfargs,{user_sup,start,[]}},{restart_type,temporary},{shutdown,2000},{child_type,supervisor}]

To be able to use rabbitmqctl I had to kill the processes and start the servers again but this is not ideal.

Thank you.

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

We haven't seen this in a few years but the trace is specific enough.

user_sup is a part of Erlang/OTP, not RabbitMQ. It is responsible for certain I/O operations and if it fails for any reason, it will affect command line tools first and foremost. You have to restart the node (the runtime process) using service or similar (depending on how RabbitMQ was installed). There is no way to recover from a failed user process in the runtime.

[michaelklishin]

It's worth mentioning that the node likely is running but CLI tools run into an error because of this failing/effectively missing I/O process that leads them to believe that RabbitMQ must have been stopped on this node.