Improve explicit histogram contention performance

[jack-berg]

My goal is to have OpenTelemetry metrics equal or exceed performance of other metrics systems, in an apples-to-apples comparison.

In this PR, I address the poor performance of explicit histograms under contention, as embodied by this prometheus/client_java benchmark.

I have issues with this benchmark. The real performance characteristic is more nuanced than can be captured by recording to a single bound series. To get the complete picture, I extended MetricRecordBenchmark to add a prometheus=false|true parameter, allowing comparison against prometheus alongside the parameters we've deemed valuable.

But there's no denying it: otel java's current simple lock based approach (predictably) gets steamrolled by a lock free design under contention.

The problem with fixing this is that lock based approach performs maximally well without contention. I've been thinking / iterating on designs on and off for months and there is no change I've found or am aware of that improves contended performance without regressing uncontended performance. This has been a sticking point in the past. So the approach I've taken with this PR:

• Thoroughly explore the solution space and to minimize the uncontended regression while still achieving the desired contended performance. I've explored dozens and dozens of angles and believe this will be difficult to improve on. The implementation in this PR is a version of the lock free design in prometheus/client_java, with modifications to avoid the complexity of having a buffer and to also optionally record min and max.
• Couple the regression with an improvement to partially offset. Observed that looking up Context.current() when exemplars are disabled is unnecessary and represents a significant chunk of work (at least on a relative basis).
• Plan for an escape hatch. I recommend we proceed with this implementation and accept the regressed uncontended performance. If users report the uncontended regression is unacceptable for their workloads, we can add a lock-based-implementation opt-in via ExplicitBucketHistogramOptions. Not adding preemptively without demand.

So how do we compare to prometheus/client_java? There are a lot of params that make it a bit overwhelming to interpret the data. Many of the param combinations tell variations of the same overall story. I've compressed the story into one graph which I think is fair and understandable. Notes:

• Light lines are before (i.e. main), dark lines are after (i.e. the PR branch). Prometheus only has after series because it's not being modified.
• Dashed lines are uncontended (i.e. 1 thread), solid lines are contended (i.e. 4 threads).
• Orange lines are prometheus, purple lines are OpenTelemetry.
• Benchmark includes these changes, which includes prometheus, disables capturing min/max for apples-to-apples comparison, and extending cardinality param set from [1, 128] to [1, 4, 32, 128] (this makes it easier to see how performance generally falls out as a function of cardinality).
• All data is available here, but the graph only includes cases where temporality=cumulative, bound=true.

Below is a before and after of the unmodified MetricRecordBenchmark. This shows clearly shows the contended gains and uncontended regressions.

Details

Benchmark	Threads	Temporality	bound	Cardinality	Instrument	Baseline (ops/s)	After (ops/s)	Δ ops/s	Δ %
threads1	1	CUMULATIVE	false	1	COUNTER_SUM	159,174,603	181,003,526	+21,828,923	+13.7%
threads1	1	CUMULATIVE	false	1	GAUGE_LAST_VALUE	180,421,620	220,676,601	+40,254,981	+22.3%
threads1	1	CUMULATIVE	false	1	HISTOGRAM_BASE2_EXPONENTIAL	39,564,541	39,159,258	-405,283	-1.0%
threads1	1	CUMULATIVE	false	1	HISTOGRAM_EXPLICIT	98,346,318	76,852,782	-21,493,536	-21.9%
threads1	1	CUMULATIVE	false	1	UP_DOWN_COUNTER_SUM	159,273,141	183,478,744	+24,205,603	+15.2%
threads1	1	CUMULATIVE	false	128	COUNTER_SUM	112,037,555	157,638,779	+45,601,224	+40.7%
threads1	1	CUMULATIVE	false	128	GAUGE_LAST_VALUE	137,759,144	209,801,648	+72,042,504	+52.3%
threads1	1	CUMULATIVE	false	128	HISTOGRAM_BASE2_EXPONENTIAL	41,958,634	47,406,931	+5,448,298	+13.0%
threads1	1	CUMULATIVE	false	128	HISTOGRAM_EXPLICIT	69,038,830	62,387,922	-6,650,908	-9.6%
threads1	1	CUMULATIVE	false	128	UP_DOWN_COUNTER_SUM	103,541,746	157,033,115	+53,491,369	+51.7%
threads1	1	CUMULATIVE	true	1	COUNTER_SUM	216,284,678	220,589,798	+4,305,120	+2.0%
threads1	1	CUMULATIVE	true	1	GAUGE_LAST_VALUE	287,609,391	382,904,194	+95,294,803	+33.1%
threads1	1	CUMULATIVE	true	1	HISTOGRAM_BASE2_EXPONENTIAL	52,279,769	55,082,188	+2,802,419	+5.4%
threads1	1	CUMULATIVE	true	1	HISTOGRAM_EXPLICIT	127,720,648	127,870,799	+150,150	+0.1%
threads1	1	CUMULATIVE	true	1	UP_DOWN_COUNTER_SUM	217,187,604	221,729,268	+4,541,664	+2.1%
threads1	1	CUMULATIVE	true	128	COUNTER_SUM	188,757,472	196,226,828	+7,469,357	+4.0%
threads1	1	CUMULATIVE	true	128	GAUGE_LAST_VALUE	267,596,664	303,244,562	+35,647,898	+13.3%
threads1	1	CUMULATIVE	true	128	HISTOGRAM_BASE2_EXPONENTIAL	49,473,201	53,507,107	+4,033,906	+8.2%
threads1	1	CUMULATIVE	true	128	HISTOGRAM_EXPLICIT	94,697,876	78,348,677	-16,349,198	-17.3%
threads1	1	CUMULATIVE	true	128	UP_DOWN_COUNTER_SUM	194,386,620	213,410,207	+19,023,588	+9.8%
threads1	1	DELTA	false	1	COUNTER_SUM	103,060,887	132,780,210	+29,719,323	+28.8%
threads1	1	DELTA	false	1	GAUGE_LAST_VALUE	115,126,942	157,113,823	+41,986,880	+36.5%
threads1	1	DELTA	false	1	HISTOGRAM_BASE2_EXPONENTIAL	42,251,451	46,941,291	+4,689,840	+11.1%
threads1	1	DELTA	false	1	HISTOGRAM_EXPLICIT	69,968,545	63,730,112	-6,238,433	-8.9%
threads1	1	DELTA	false	1	UP_DOWN_COUNTER_SUM	100,604,149	134,571,855	+33,967,706	+33.8%
threads1	1	DELTA	false	128	COUNTER_SUM	94,996,648	104,334,513	+9,337,865	+9.8%
threads1	1	DELTA	false	128	GAUGE_LAST_VALUE	103,919,545	137,142,661	+33,223,116	+32.0%
threads1	1	DELTA	false	128	HISTOGRAM_BASE2_EXPONENTIAL	40,857,565	43,124,499	+2,266,933	+5.5%
threads1	1	DELTA	false	128	HISTOGRAM_EXPLICIT	65,982,406	54,999,057	-10,983,349	-16.6%
threads1	1	DELTA	false	128	UP_DOWN_COUNTER_SUM	89,948,202	106,396,894	+16,448,692	+18.3%
threads1	1	DELTA	true	1	COUNTER_SUM	191,164,825	217,047,305	+25,882,479	+13.5%
threads1	1	DELTA	true	1	GAUGE_LAST_VALUE	219,574,947	241,560,171	+21,985,224	+10.0%
threads1	1	DELTA	true	1	HISTOGRAM_BASE2_EXPONENTIAL	43,679,523	51,232,168	+7,552,645	+17.3%
threads1	1	DELTA	true	1	HISTOGRAM_EXPLICIT	112,259,822	115,575,686	+3,315,864	+3.0%
threads1	1	DELTA	true	1	UP_DOWN_COUNTER_SUM	193,173,732	216,559,730	+23,385,998	+12.1%
threads1	1	DELTA	true	128	COUNTER_SUM	149,534,516	161,139,519	+11,605,003	+7.8%
threads1	1	DELTA	true	128	GAUGE_LAST_VALUE	199,519,442	212,047,879	+12,528,437	+6.3%
threads1	1	DELTA	true	128	HISTOGRAM_BASE2_EXPONENTIAL	46,669,711	50,867,121	+4,197,410	+9.0%
threads1	1	DELTA	true	128	HISTOGRAM_EXPLICIT	78,538,092	67,602,837	-10,935,255	-13.9%
threads1	1	DELTA	true	128	UP_DOWN_COUNTER_SUM	153,385,372	161,985,188	+8,599,815	+5.6%
threads4	4	CUMULATIVE	false	1	COUNTER_SUM	514,286,822	708,503,743	+194,216,921	+37.8%
threads4	4	CUMULATIVE	false	1	GAUGE_LAST_VALUE	80,892,277	142,657,590	+61,765,313	+76.4%
threads4	4	CUMULATIVE	false	1	HISTOGRAM_BASE2_EXPONENTIAL	18,659,983	22,262,832	+3,602,850	+19.3%
threads4	4	CUMULATIVE	false	1	HISTOGRAM_EXPLICIT	16,256,760	222,388,385	+206,131,625	+1268.0%
threads4	4	CUMULATIVE	false	1	UP_DOWN_COUNTER_SUM	522,668,485	651,879,434	+129,210,949	+24.7%
threads4	4	CUMULATIVE	false	128	COUNTER_SUM	219,682,318	269,367,685	+49,685,367	+22.6%
threads4	4	CUMULATIVE	false	128	GAUGE_LAST_VALUE	194,449,854	200,100,196	+5,650,341	+2.9%
threads4	4	CUMULATIVE	false	128	HISTOGRAM_BASE2_EXPONENTIAL	76,961,290	76,902,492	-58,798	-0.1%
threads4	4	CUMULATIVE	false	128	HISTOGRAM_EXPLICIT	83,043,913	107,603,810	+24,559,897	+29.6%
threads4	4	CUMULATIVE	false	128	UP_DOWN_COUNTER_SUM	197,822,741	265,230,790	+67,408,049	+34.1%
threads4	4	CUMULATIVE	true	1	COUNTER_SUM	760,947,556	806,169,105	+45,221,549	+5.9%
threads4	4	CUMULATIVE	true	1	GAUGE_LAST_VALUE	106,458,188	118,441,059	+11,982,871	+11.3%
threads4	4	CUMULATIVE	true	1	HISTOGRAM_BASE2_EXPONENTIAL	15,994,875	21,967,870	+5,972,995	+37.3%
threads4	4	CUMULATIVE	true	1	HISTOGRAM_EXPLICIT	20,207,252	175,771,793	+155,564,540	+769.8%
threads4	4	CUMULATIVE	true	1	UP_DOWN_COUNTER_SUM	755,569,812	791,905,411	+36,335,599	+4.8%
threads4	4	CUMULATIVE	true	128	COUNTER_SUM	309,793,280	425,498,504	+115,705,224	+37.3%
threads4	4	CUMULATIVE	true	128	GAUGE_LAST_VALUE	147,477,523	141,623,611	-5,853,912	-4.0%
threads4	4	CUMULATIVE	true	128	HISTOGRAM_BASE2_EXPONENTIAL	79,865,307	79,652,866	-212,441	-0.3%
threads4	4	CUMULATIVE	true	128	HISTOGRAM_EXPLICIT	84,537,257	125,412,347	+40,875,090	+48.4%
threads4	4	CUMULATIVE	true	128	UP_DOWN_COUNTER_SUM	306,769,934	463,678,234	+156,908,300	+51.1%
threads4	4	DELTA	false	1	COUNTER_SUM	34,632,977	34,389,311	-243,666	-0.7%
threads4	4	DELTA	false	1	GAUGE_LAST_VALUE	17,034,642	15,870,071	-1,164,571	-6.8%
threads4	4	DELTA	false	1	HISTOGRAM_BASE2_EXPONENTIAL	11,379,075	12,921,303	+1,542,228	+13.6%
threads4	4	DELTA	false	1	HISTOGRAM_EXPLICIT	13,635,919	28,757,167	+15,121,248	+110.9%
threads4	4	DELTA	false	1	UP_DOWN_COUNTER_SUM	34,269,374	32,308,628	-1,960,746	-5.7%
threads4	4	DELTA	false	128	COUNTER_SUM	90,367,440	94,321,869	+3,954,429	+4.4%
threads4	4	DELTA	false	128	GAUGE_LAST_VALUE	95,927,762	96,807,237	+879,474	+0.9%
threads4	4	DELTA	false	128	HISTOGRAM_BASE2_EXPONENTIAL	57,688,448	55,023,965	-2,664,483	-4.6%
threads4	4	DELTA	false	128	HISTOGRAM_EXPLICIT	59,735,171	76,044,837	+16,309,665	+27.3%
threads4	4	DELTA	false	128	UP_DOWN_COUNTER_SUM	94,511,477	92,610,908	-1,900,569	-2.0%
threads4	4	DELTA	true	1	COUNTER_SUM	37,955,800	35,011,485	-2,944,315	-7.8%
threads4	4	DELTA	true	1	GAUGE_LAST_VALUE	20,848,949	20,848,785	-164	-0.0%
threads4	4	DELTA	true	1	HISTOGRAM_BASE2_EXPONENTIAL	14,334,174	13,327,637	-1,006,536	-7.0%
threads4	4	DELTA	true	1	HISTOGRAM_EXPLICIT	14,183,117	24,715,450	+10,532,333	+74.3%
threads4	4	DELTA	true	1	UP_DOWN_COUNTER_SUM	38,227,294	37,857,357	-369,937	-1.0%
threads4	4	DELTA	true	128	COUNTER_SUM	95,074,745	95,397,929	+323,183	+0.3%
threads4	4	DELTA	true	128	GAUGE_LAST_VALUE	88,510,456	82,493,838	-6,016,618	-6.8%
threads4	4	DELTA	true	128	HISTOGRAM_BASE2_EXPONENTIAL	57,774,179	57,575,826	-198,353	-0.3%
threads4	4	DELTA	true	128	HISTOGRAM_EXPLICIT	58,979,504	80,909,002	+21,929,498	+37.2%
threads4	4	DELTA	true	128	UP_DOWN_COUNTER_SUM	93,802,234	94,691,464	+889,230	+0.9%

Takeaways from a performance standpoint:

• Huge breakthrough in contended explicit histogram performance, ranging from 27%-1268%, with larger gains with low cardinality. I believe larger cardinality changes the cache profile. I.e. L1 vs. L2.
• Regression in uncontended cases ranging from 9%-22%. Although a few are flatlined.
• With this change, OpenTelemetry catches up to (and actually exceeds but prometheus could easily adapt to match) prometheus for contended performance, while equaling it for uncontended performance. For the cardinality=1, bound=true case the prometheus benchmark focuses on, OpenTelemetry improves from 23.3->224.6 million ops/s.

[opentelemetry-pr-dashboard]

Pull request dashboard status

Waiting on reviewers · refreshed 2026-08-10 21:20 UTC

Review the latest changes.

Status above doesn't look right?

• Just replied or pushed? Anything around or after the refresh time above may not be picked up yet — give it a few minutes.
• Anything look wrong? Report it with what you expected; it helps us improve the dashboard.

[jack-berg]

This new test checks that the intermediate state of every collect is correct and doesn't include partial writes. Asserting this was particularly important with the complex lock free implementations I experimented with. The current test asserts that the aggregate state after a bunch of records and collects is correct, which ensures no lost writes or double writes.

But we had no tests asserting no partial writes, and with a lock based implementation it was easier to ignore this lack of coverage.

[codecov]

Codecov Report

❌ Patch coverage is 93.16239% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.65%. Comparing base (dd811c4) to head (4ecc89e).
⚠️ Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...gator/DoubleExplicitBucketHistogramAggregator.java	92.13%	1 Missing and 6 partials ⚠️
.../opentelemetry/sdk/metrics/AbstractInstrument.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8717      +/-   ##
============================================
+ Coverage     91.62%   91.65%   +0.02%     
- Complexity    10327    10353      +26     
============================================
  Files          1003     1003              
  Lines         27138    27267     +129     
  Branches       3187     3220      +33     
============================================
+ Hits          24866    24991     +125     
- Misses         1566     1567       +1     
- Partials        706      709       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.