Per-queue Raft metrics

[paguos]

Is your feature request related to a problem? Please describe.

I would like to have queue base metrics to see the raft status of each Quorum Queue.

Describe the solution you'd like

Add additional metrics to monitor the status of the quorum queues. The main need for this is to be able to create alerts when a queue's quorum is under a given limit.

Describe alternatives you've considered

I try to get that information from existing metrics:

curl localhost:15692/metrics | grep raft
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 98950  100 98950    0     0   160k      0 --:--:-- --:--:-- --:--:--  160k
# TYPE rabbitmq_raft_term_total counter
# HELP rabbitmq_raft_term_total Current Raft term number
rabbitmq_raft_term_total 1808
# TYPE rabbitmq_raft_log_snapshot_index gauge
# HELP rabbitmq_raft_log_snapshot_index Raft log snapshot index
rabbitmq_raft_log_snapshot_index 57764232
# TYPE rabbitmq_raft_log_last_applied_index gauge
# HELP rabbitmq_raft_log_last_applied_index Raft log last applied index
rabbitmq_raft_log_last_applied_index 57780704
# TYPE rabbitmq_raft_log_commit_index gauge
# HELP rabbitmq_raft_log_commit_index Raft log commit index
rabbitmq_raft_log_commit_index 57780704
# TYPE rabbitmq_raft_log_last_written_index gauge
# HELP rabbitmq_raft_log_last_written_index Raft log last written index
rabbitmq_raft_log_last_written_index 57780704
# TYPE rabbitmq_raft_entry_commit_latency_seconds gauge
# HELP rabbitmq_raft_entry_commit_latency_seconds Time taken for a log entry to be committed
rabbitmq_raft_entry_commit_latency_seconds 0.0023636363636363638

But none of them have any label that can be used for getting the quorum of an individual label.

Additional context

No response

[michaelklishin]

rabbitmq-queues quorum_status

already provides what can be provided per-queue.

There are already over 60 or so metrics per queue, even after a few of them were removed in 4.0.x.
This can and in practice will create significant cardinality issues when visualizing them, which is
why for years now the Prometheus plugin has had two operating modes and scraping endpoints. You are using the aggregated one.

[paguos]

That explains a lot. Thank you 👍🏼

[michaelklishin]

@paguos this should have been a discussion. You are likely unaware of this section of the Prometheus plugin documentation.

Plus there are #12148 and #13107, so, evidence of new Raft metrics being added over time for QQs and other Raft-based components such as Khepri.

[paguos]

Thanks for the feedback. Agree this should be a discussion. I was not aware of that feature. Thanks for the help!

[paguos]

@michaelklishin could you help pointing out the metrics I need to use to have the following information in a grafana chart?

I'm interested in the number of members of a quorum for each queue.

[michaelklishin]

@paguos I don't know. That UI element uses the queues.{n}.online array:

function fmt_members(queue) {
    var res = '';
    var isMajority = (queue.online.length >= (Math.floor(queue.members.length / 2) + 1));
    var followers = queue.online;
    followers.splice(followers.indexOf(queue.node), 1);
    if (isMajority) {
        res += ' <abbr title="Followers: ' + followers + '">+' +
            followers.length + '</abbr>';
    } else {
        res += ' <abbr class="warning" title="Cluster is in minority">+' + followers.length + '</abbr>';
    }
    return res;
}

You'll have to find a Prometheus endpoint or a Grafana expression equivalent, or contribute them if they do not exist. This is open source software after all.

[michaelklishin]

@mkuratczyk might help, he is looking into how we can improve the Grafana dashboards