You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: proposals/0048-otel_delta_temporality_support.md
+32-30Lines changed: 32 additions & 30 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -102,6 +102,8 @@ These may come in later iterations of delta support.
102
102
103
103
## How
104
104
105
+
This section outlines the proposed approach for delta temporality support. Additional approaches and extensions can be found in [Potential future extensions](#potential-future-extensions) and [Discarded alternatives](#discarded-alternatives).
106
+
105
107
### Ingesting deltas
106
108
107
109
When an OTLP sample has its aggregation temporality set to delta, write its value at `TimeUnixNano`.
@@ -261,6 +263,34 @@ There are also additional functions that should only be used with Prometheus gau
261
263
262
264
This initial approach enables Prometheus to support OTEL delta metrics in a careful manner by relying on existing concepts like labels and reusing existing functions with minimal changes, and also provides a foundation for potentially building more advanced delta features in the future.
263
265
266
+
## Known unknowns
267
+
268
+
### Native histograms performance
269
+
270
+
To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time.
271
+
272
+
## Implementation Plan
273
+
274
+
### 1. Experimental feature flags for OTLP delta ingestion
275
+
276
+
Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion.
277
+
278
+
Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any type metadata to the ingested series.
279
+
280
+
### 2. Function warnings
281
+
282
+
Add function warnings when a function is used with series of wrong type or temporality as described in [Function warnings](#function-warnings).
283
+
284
+
There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632.
285
+
286
+
### 3. Update documentation
287
+
288
+
Update documentation explaining new experimental delta functionality, including recommended configs for filtering out delta metrics from scraping and remote write.
289
+
290
+
### 4. Review deltas in practice and experiment with possible future extensions
291
+
292
+
Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label).
@@ -303,7 +333,7 @@ The `smoothed` modifer in the extended range selectors proposal does this for cu
303
333
304
334
The `smoothed` proposal works by injecting points at the edges of the range. For the start boundary, the injected point will have its value worked out by linearly interpolating between the closest point before the range start and the first point inside the range.
That value would be nonesensical for deltas, as the values for delta samples are independent. Additionally, for deltas, to work out the increase, we add all the values up in the range (with some adjustments) vs in the cumulative case where you subtract the first point in the range from the last point. So it makes sense the smoothing behaviour would be different.
309
339
@@ -446,32 +476,4 @@ Delta to cumulative conversion at query time doesn’t have the same out of orde
446
476
447
477
No function modification is needed - all cumulative functions will work for samples ingested as deltas.
448
478
449
-
However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions.
450
-
451
-
## Known unknowns
452
-
453
-
### Native histograms performance
454
-
455
-
To work out the delta for all the cumulative native histograms in an range, the first sample is subtracted from the last and then adjusted for counter resets within all the samples. Counter resets are detected at ingestion time when possible. This means the query engine does not have to read all buckets from all samples to calculate the result. The same is not true for delta metrics - as each sample is independent, to get the delta between the start and end of the range, all of the buckets in all of the samples need to be summed, which is less efficient at query time.
456
-
457
-
## Implementation Plan
458
-
459
-
### 1. Experimental feature flags for OTLP delta ingestion
460
-
461
-
Add the `--enable-feature=otlp-native-delta-ingestion` and `--enable-feature=otlp-delta-as-gauge-ingestion` feature flags as described in [Delta metric type](#delta-metric-type) to add appropriate types and flags to series on ingestion.
462
-
463
-
Note a `--enable-feature=otlp-native-delta-ingestion` was already introduced in https://github.com/prometheus/prometheus/pull/16360, but that doesn't add any types to deltas.
464
-
465
-
### 2. Function warnings
466
-
467
-
Add function warnings when a function is used with series of wrong type or temporality as described in [Function warnings](#function-warnings).
468
-
469
-
There are already warnings if `rate()`/`increase()` are used without the `__type__="counter"` label: https://github.com/prometheus/prometheus/pull/16632.
470
-
471
-
### 3. Update documentation
472
-
473
-
Update documentation explaining new experimental delta functionality, including recommended configs for filtering out delta metrics from scraping and remote write.
474
-
475
-
### 4. Review deltas in practice and experiment with possible future extensions
476
-
477
-
Review how deltas work in practice using the current approach, and use experience and feedback to decide whether any of the potential extensions are necessary, and whether to discontinue one of the two options for delta ingestion (gauges vs. temporality label).
479
+
However, it can be confusing for users that the delta samples they write are transformed into cumulative samples with different values during querying. The sparseness of delta metrics also do not work well with the current `rate()` and `increase()` functions.
0 commit comments