Skip to content

Commit 0f1a701

Browse files
committed
mixin: longer rate interval for Alertmanager[Cluster]FailedToSendAlerts
5m is an unfortunate value since its the same as the default group_interval. If only a single alert fires and a single notifier fails consistently the alert will never fire, since the rate[5m] will be 0 after less then the group_interval. Signed-off-by: Jan Fajerski <[email protected]>
1 parent 6703b0a commit 0f1a701

File tree

1 file changed

+6
-6
lines changed

1 file changed

+6
-6
lines changed

doc/alertmanager-mixin/alerts.libsonnet

+6-6
Original file line numberDiff line numberDiff line change
@@ -42,9 +42,9 @@
4242
alert: 'AlertmanagerFailedToSendAlerts',
4343
expr: |||
4444
(
45-
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s}[5m])
45+
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s}[15m])
4646
/
47-
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[5m])
47+
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s}[15m])
4848
)
4949
> 0.01
5050
||| % $._config,
@@ -61,9 +61,9 @@
6161
alert: 'AlertmanagerClusterFailedToSendAlerts',
6262
expr: |||
6363
min by (%(alertmanagerClusterLabels)s, integration) (
64-
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s, integration=~`%(alertmanagerCriticalIntegrationsRegEx)s`}[5m])
64+
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s, integration=~`%(alertmanagerCriticalIntegrationsRegEx)s`}[15m])
6565
/
66-
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s, integration=~`%(alertmanagerCriticalIntegrationsRegEx)s`}[5m])
66+
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s, integration=~`%(alertmanagerCriticalIntegrationsRegEx)s`}[15m])
6767
)
6868
> 0.01
6969
||| % $._config,
@@ -80,9 +80,9 @@
8080
alert: 'AlertmanagerClusterFailedToSendAlerts',
8181
expr: |||
8282
min by (%(alertmanagerClusterLabels)s, integration) (
83-
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s, integration!~`%(alertmanagerCriticalIntegrationsRegEx)s`}[5m])
83+
rate(alertmanager_notifications_failed_total{%(alertmanagerSelector)s, integration!~`%(alertmanagerCriticalIntegrationsRegEx)s`}[15m])
8484
/
85-
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s, integration!~`%(alertmanagerCriticalIntegrationsRegEx)s`}[5m])
85+
ignoring (reason) group_left rate(alertmanager_notifications_total{%(alertmanagerSelector)s, integration!~`%(alertmanagerCriticalIntegrationsRegEx)s`}[15m])
8686
)
8787
> 0.01
8888
||| % $._config,

0 commit comments

Comments
 (0)