feat: Add new `tier/used/capacity` metric to coordinator that reports actual loaded bytes across a tier by capistrant · Pull Request #20199 · apache/druid

capistrant · 2026-08-31T15:00:50Z

Description

new metric `tier/storage/used`

tier/storage/used is a new coordinator metric that emits with a tier and optional tierAlias (if you enabled aliasing) dimension. This metric is helpful for monitoring the actually realized loaded bytes on a tier. Especially in a world of partial loads, where your tier may "require" a very high amount of storage to store everything on disk as a full load, but if you are using virtual storage and partial loads, you care more about the actual realized bytes on your historicals and may want to monitor this true consumption.

renamed metric `tier/total/capacity` to `tier/assignable/capacity`

old metric will continue to be emitted until druid 40. The rename is being done because the value is not a true real capacity but rather assignable in that with virtual storage you can over assign to a tier because virtual storage will evict segments if it needs to load others.

document `tier/storage/capacity`

this metric was added but not documented. it is the real physical capacity of the tier.

statsd and prometheus

the default metrics are updated to reflect the new metrics as well as emit the existing but missing tier/storage/capacity which is the true physical size.

Release note

New coordinator metric tier/storage/used that helps track the actual bytes used per tier in a cluster. deprecated tier/total/capacity and replaced it with tier/assignable/capacity. both will be emitted, but please move your monitors to the new metric before druid 40 releases.

Key changed/added classes in this PR

PrepareBalancerAndLoadQueues

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

kfaraz

Minor suggestion on naming.

kfaraz · 2026-08-31T16:26:00Z

 |`tier/replication/factor`|Configured maximum replication factor in each tier. The `tierAlias` dimension is emitted only when the tier belongs to an alias configured via [`historicalTierAliases`](../configuration/index.md#dynamic-configuration).|`tier`, `tierAlias`|Varies|
 |`tier/required/capacity`|Total capacity in bytes required in each tier. The `tierAlias` dimension is emitted only when the tier belongs to an alias configured via [`historicalTierAliases`](../configuration/index.md#dynamic-configuration).|`tier`, `tierAlias`|Varies|
-|`tier/total/capacity`|Total capacity in bytes available in each tier. The `tierAlias` dimension is emitted only when the tier belongs to an alias configured via [`historicalTierAliases`](../configuration/index.md#dynamic-configuration).|`tier`, `tierAlias`|Varies|
+|`tier/total/capacity`|Total capacity in bytes available in each tier. Can be greater than physical disk capacity if using virtual storage. The `tierAlias` dimension is emitted only when the tier belongs to an alias configured via [`historicalTierAliases`](../configuration/index.md#dynamic-configuration).|`tier`, `tierAlias`|Varies|


With virtual storage, I feel total/capacity could now be ambiguous. Does this metric currently (before this PR) report the total virtual capacity or the total physical capacity?

Since we are revisiting these metrics, I feel we could distinguish them by naming them something like:

tier/storage/required

tier/storage/physicalCapacity

tier/storage/virtualCapacity

tier/storage/used

and deprecated the existing tier/total/capacity metric.

tier/storage/used I fully agree with. that is better name than I had, wil change.

I am considering something like tier/storage/assignableCapacity as the replacement for deprecated tier/total/capacity. It is what the coordinator sees as the amount of data it can assign to a tier.

then leaving tier/storage/capacity as is to reduce metric name churn?

thoughts?

I guess I forgot about required in this calculus of trying to reduce churn

ok my latest thought is

tier/required/capacity (unchanged) rule demand for full loads tier/assignable/capacity (renamed) sum(maxSize) — assignment budget tier/storage/capacity (newly documented) sum(storageSize) — real disk tier/storage/used (new) sum(currSize) — real bytes on disk

With the tier/storage, I was trying to ensure that they all have the same metric prefix.
But your suggested names would work too 👍🏻 , as long as it is easy to distinguish between the physical and the virtual.

kfaraz

Minor suggestions, changes look good.

kfaraz · 2026-09-01T11:39:48Z

+    // Superseded by ASSIGNABLE_CAPACITY, which carries the same value under a name that says it is an assignment
+    // budget rather than physical disk. Still emitted so existing dashboards keep working; remove after deprecation.


Please convert this into a javadoc.

kfaraz · 2026-09-01T11:40:08Z

  public static class Tier
  {
+    // Assignment budget of a tier, denominated in full segment size. REQUIRED_CAPACITY is the demand the rules
+    // place on the tier to full load all segments, ASSIGNABLE_CAPACITY the supply the tier advertises for it. Both are


Please move the part about the assignable capacity into a javadoc for that field itself.

kfaraz · 2026-09-01T11:40:19Z


  public static class Tier
  {
+    // Assignment budget of a tier, denominated in full segment size. REQUIRED_CAPACITY is the demand the rules


Please convert to javadoc

kfaraz · 2026-09-01T11:41:39Z

+  /**
+   * {@code tier/storage/used} reports what the tier's historicals have announced as loaded, so it stays at zero
+   * until segments are actually on disk, unlike {@code tier/required/capacity} which reports what the rules demand.
+   */


Nit: Try to remove this javadoc and incorporate what it does into the method name itself, and small 1-line comments inside the method, if needed.

FrankChen021

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Severity	Findings
P0	0
P1	0
P2	1
P3	0
Total	1

Reviewed 5 of 5 changed files.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-09-01T12:25:38Z

    public static final CoordinatorStat REQUIRED_CAPACITY
        = CoordinatorStat.toDebugAndEmit("reqdCap", "tier/required/capacity");
+    public static final CoordinatorStat ASSIGNABLE_CAPACITY
+        = CoordinatorStat.toDebugAndEmit("assignableCap", "tier/assignable/capacity");


[P2] Register new tier metrics in default emitter maps

The new tier/assignable/capacity and tier/storage/used stats are absent from the bundled Prometheus, StatsD, and filtered logging metric maps, so default exporters omit these metrics even though the coordinator emits them. Add both metrics with their tier and tierAlias dimensions and gauge semantics, plus mapping coverage tests.

…n this PR

FrankChen021

Severity	Findings
P0	0
P1	1
P2	0
P3	0
Total	1

Reviewed 7 of 7 changed files.

Validation: reverse git apply --check against the canonical diff passed. Builds and tests were not run.

This is an automated review by Codex GPT-5.6-Luna(max)

FrankChen021 · 2026-09-02T12:31:27Z

  "tier/replication/factor" : { "dimensions" : ["tier", "tierAlias"], "type" : "count", "help": "Configured maximum replication factor in each tier."},
  "tier/required/capacity" : { "dimensions" : ["tier", "tierAlias"], "type" : "count", "help": "Total capacity in bytes required in each tier."},
  "tier/total/capacity" : { "dimensions" : ["tier", "tierAlias"], "type" : "count", "help": "Total capacity in bytes available in each tier."},
+  "tier/assignable/capacity": { "dimensions" : ["tier", "tierAlias"], "type" : "count", "help": "Total capacity in bytes available for assignment in each tier."},


[P1] Register tier state metrics as gauges

The new tier/assignable/capacity, tier/storage/capacity, and tier/storage/used metrics are configured as Prometheus counters. PrometheusEmitter increments counters for every coordinator event, so these current-state values accumulate across cycles, never decrease when capacity or usage falls, and are exposed with counter semantics. Configure all three as gauge metrics.

Minor historical server disk usage metric changes to help in monitoring

eacfa27

github-actions Bot added the Area - Documentation label Aug 31, 2026

kfaraz reviewed Aug 31, 2026

View reviewed changes

capistrant added 5 commits August 31, 2026 13:01

rename metric

bb2dd26

rename metric and deprecate old name

3f5823c

remove extra doc context that is unhelpful

5428617

wording

6779d85

another potential iteration

884d7d3

kfaraz approved these changes Sep 1, 2026

View reviewed changes

FrankChen021 reviewed Sep 1, 2026

View reviewed changes

capistrant added 2 commits September 1, 2026 11:35

tidy up metric docs

0b7af6f

get the prometheus and statsd metrics maps updated with the changes i…

a93d930

…n this PR

github-actions Bot added the Area - Metrics/Event Emitting label Sep 1, 2026

capistrant added 2 commits September 1, 2026 14:12

Merge branch 'master' into partial-stats

608c8ac

Merge branch 'master' into partial-stats

c686511

FrankChen021 reviewed Sep 2, 2026

View reviewed changes

		// Superseded by ASSIGNABLE_CAPACITY, which carries the same value under a name that says it is an assignment
		// budget rather than physical disk. Still emitted so existing dashboards keep working; remove after deprecation.

Uh oh!

Conversation

capistrant commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

new metric tier/storage/used

renamed metric tier/total/capacity to tier/assignable/capacity

document tier/storage/capacity

statsd and prometheus

Release note

Key changed/added classes in this PR

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

capistrant Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

capistrant commented Aug 31, 2026 •

edited

Loading

new metric `tier/storage/used`

renamed metric `tier/total/capacity` to `tier/assignable/capacity`

document `tier/storage/capacity`

capistrant Sep 1, 2026 •

edited

Loading