Skip to content

Add per-tenant Grafana Explore URL format for alert GeneratorURL#7302

Open
CharlieTLe wants to merge 2 commits intocortexproject:masterfrom
CharlieTLe:809
Open

Add per-tenant Grafana Explore URL format for alert GeneratorURL#7302
CharlieTLe wants to merge 2 commits intocortexproject:masterfrom
CharlieTLe:809

Conversation

@CharlieTLe
Copy link
Member

Summary

  • Add support for tenants to configure alert GeneratorURL to use Grafana Explore format instead of the default Prometheus /graph format
  • Introduce three new per-tenant runtime config settings: ruler_alert_generator_url_format, ruler_grafana_datasource_uid, and ruler_grafana_org_id
  • Add per-tenant ruler_external_url override so tenants can have different external URLs without changing the global ruler config

When ruler_alert_generator_url_format is set to grafana-explore, the GeneratorURL in fired alerts becomes a Grafana Explore link (e.g. http://grafana:3000/explore?schemaVersion=1&panes=...&orgId=1) with the alert expression and datasource UID pre-filled. Clicking "See source" on an alert in Grafana opens Explore with the correct query, datasource, and time range.

New per-tenant config options

Setting Description Default
ruler_external_url Per-tenant external URL override for the ruler "" (uses global)
ruler_alert_generator_url_format URL format: prometheus or grafana-explore "" (prometheus)
ruler_grafana_datasource_uid Grafana datasource UID for Explore links ""
ruler_grafana_org_id Grafana org ID for Explore links 1

Example runtime config

overrides:
  my-tenant:
    ruler_external_url: "http://grafana.example.com:3000"
    ruler_alert_generator_url_format: "grafana-explore"
    ruler_grafana_datasource_uid: "my-cortex-ds"
    ruler_grafana_org_id: 1

Test plan

  • Unit tests for grafanaExploreLink URL generation with various expressions and special characters
  • Unit tests for userExternalURL tracking per-tenant URL changes
  • Unit tests for SendAlerts with custom generator URL function
  • Unit tests for ruler manager factory wiring the per-tenant overrides
  • Validated exporter test updated for new limit fields
  • Manual e2e test: Docker Compose stack with Cortex + Grafana, always-firing alert rule, runtime config with grafana-explore format — confirmed the "See source" link opens Grafana Explore with correct datasource and query

🤖 Generated with Claude Code

Add support for tenants to configure alert GeneratorURL to use Grafana
Explore format instead of the default Prometheus /graph format. This is
controlled by three new per-tenant settings: ruler_alert_generator_url_format,
ruler_grafana_datasource_uid, and ruler_grafana_org_id.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>
@dosubot dosubot bot added the component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. label Feb 27, 2026
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Charlie Le <charlie_le@apple.com>
@CharlieTLe CharlieTLe linked an issue Mar 1, 2026 that may be closed by this pull request
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajagopalanand Can you help take a look?

Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I gave a review, but I think we do not care what parameters are send to the url. We don't care about grafana org IDs

I think we should make this general enough to be used by any metrics explorer.
Maybe consider renaming the function as explore and not have any grafana

I am thinking of something that would support for example:
explore

what do you think?

cortex_overrides{limit_name="reject_old_samples",user="tenant-a"} 0
cortex_overrides{limit_name="reject_old_samples_max_age",user="tenant-a"} 1.2096e+06
cortex_overrides{limit_name="ruler_evaluation_delay_duration",user="tenant-a"} 0
cortex_overrides{limit_name="ruler_grafana_org_id",user="tenant-a"} 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should never be 0. The default org id as zero should be rejected as invalid.

Consider putting

  •   if l.RulerGrafanaOrgID < 1 {
    
  •           return errors.New("ruler_grafana_org_id must be greater than or equal to 1")
    
  •   }
    

in UnmarshalYaml and UnmarshallJSON in limits.go

Comment on lines +389 to +391
if orgID == 0 {
orgID = 1
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if orgID == 0 {
orgID = 1
}

Should not be needed after my other comment

if orgID == 0 {
orgID = 1
}
return grafanaExploreLink(externalURL, expr, datasourceUID, orgID)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think is better we make grafanaExploreLink return errors if the url cannot be generated. For example: empty datasource
and if get errors we should log the error and fallback to prometheus still


# Grafana datasource UID for alert generator URLs when format is
# grafana-explore.
[ruler_grafana_datasource_uid: <string> | default = ""]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's not clear looking at this configuration that empty is not valid.

},
},
"range": map[string]string{
"from": "now-1h",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this looks like it should be configurable. Maybe there is a way to configure the whole pane as a configurable json. At the end of the day, we don't care for this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. size/L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Alert GeneratorURLs generated by the ruler do not indicate tenant

3 participants