Skip to content

Conversation

@yeya24
Copy link
Contributor

@yeya24 yeya24 commented Dec 23, 2025

What this PR does:

This PR introduces a new feature to keep track of number of series queried on Ingester per tenant and expose it as metrics. Number of series queried can be tracked via configurable time windows so that we can keep track of number of series queried for the past 2h, 12h or 24h. This can be useful to understand query pattern and resource usage.

Here are some key points of this implementation:

  • HyperLogLog or HLL is used to keep track of the cardinality of queried series. This is a probabilistic data structure to estimate cardinality with minimum memory usage.
  • Each tenant will maintain a circular buffer of HLLs. Each HLL keeps track of the number of series queried for a given time window. If the time window size is configured to be 30m, to calculate the usage for the past 2h, we just need to merge HLLs from the past 4 time windows and estimate the final cardinality.
  • Users can configure different time windows to keep track of the metric. For example, 2h and 24h. With 30m time window, the circular buffer will have 48 HLLs in total. When calculating 2h metric, 4 HLLs are merged. When calculating 24h metric, 48 HLLs are merged. The actual implementation utilized caching as only the latest HLL is changing and previous HLLs' merged results are cached.
  • The hash of queried series is used to be added to HLLs to estimate cardinality. Sampling can be enabled via ingester.active-queried-series-metrics-sample-rate to reduce resource usage as queries go to ingesters are likely to be repetitive
  • Updating HLL with series hash is implemented using an async producer consumer pattern using channels. This avoids adding additional latency to the Ingester query path.

Which issue(s) this PR fixes:
Fixes #7151

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

@yeya24 yeya24 changed the title Add metrics for active series queried Add metrics for number of series queried Dec 23, 2025
@yeya24 yeya24 force-pushed the active-queried-series branch from cd8ee2f to 64e5235 Compare December 23, 2025 06:59
@yeya24 yeya24 force-pushed the active-queried-series branch from 64e5235 to 9c92423 Compare December 23, 2025 07:00
Signed-off-by: yeya24 <[email protected]>
Signed-off-by: yeya24 <[email protected]>
@yeya24 yeya24 force-pushed the active-queried-series branch from b3123a8 to f1ced46 Compare December 23, 2025 21:10
Signed-off-by: yeya24 <[email protected]>
@yeya24 yeya24 force-pushed the active-queried-series branch from f1ced46 to 971f4f5 Compare December 23, 2025 21:24
Copy link
Member

@alanprot alanprot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for doing this! great work! LGTM

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Dec 23, 2025
@yeya24 yeya24 merged commit 97013b9 into cortexproject:master Dec 23, 2025
48 of 51 checks passed
@yeya24 yeya24 deleted the active-queried-series branch December 23, 2025 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/ingester lgtm This PR has been approved by a maintainer size/XXL type/feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose metrics for number of active series queried within a given time period

2 participants