Report cpu_window on the top-CPU reads: the attribution denominator (#2320)

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **The top-CPU rankings hand the caller their own denominator: cpu_window** ([#2320]) - get_top_queries_by_cpu and get_top_procedures_by_cpu (both apps) now report what fraction of the SQL CPU the box ACTUALLY burned the returned rows explain: measured_sql_cpu_seconds (the collected utilization series' average x the server's core count x the window, Azure-aware via COALESCE(vcore_count, cpu_count)), attributed_cpu_seconds (the returned rows' windowed total), and attributed_ratio. Twice earned on #2235: pre-#2290 the reads explained ~10% of one production box and nothing said so, and the Datadog disagreement died the moment its worker_time sum was divided by the box's consumed CPU-seconds and produced a physically impossible 137% - this field is that division as a first-class output. A ratio under half carries a note saying where the remainder lives (below the ranking cut, plans evicted between snapshots, uncached work); a ratio past 1.1 carries the sampling-skew note - the impossibility detector. The whole object is NULL whenever the denominator cannot be trusted (fewer than three samples, a sampled span under half the window - span rather than any count-per-minute expectation, so a deliberately slowed collection cadence still qualifies - unknown core count, hard-zero average): omitted, never fabricated. Shared CpuAttribution math, decision table pinned identically in both suites.
 - **The generic webhook can now hand automation the alert's structure: `{{context_json}}`, `{{incidents_json}}` and `{{dedup_key}}`** ([#2302]) - everything automation needs already existed structured inside the product, and every channel then flattened a different half: Teams/Slack keep incident structure but bury the scalars in display strings, the generic channel keeps discrete scalars but joins the whole context into one " | " line whose delimiters collide with Victim SQL, and PagerDuty keeps only the first incident's key. The reporting consumer measured the cost precisely: 31 of 49 Logic App actions existed only to undo the flattening, including two silent-failure guesses (deriving the server by splitting the summary on " on ", detecting incident sections by substring). The new tokens are raw JSON VALUES substituted unquoted - they deliberately bypass the per-token JSON escaping, via an explicit raw set that leaves the single-pass MatchEvaluator untouched - and their shape is EXACTLY the AlertContextSerializer projection persisted as alert-history ContextJson, so a consumer parses one shape whether it reads the webhook or the history row (pinned by a round-trip test through the same serializer). `{{dedup_key}}` carries the very key the PagerDuty channel derives - including the stable serverId+metric fallback that existed in code but was never exposed to any consumer, which had forced title-matching heuristics for level/threshold alerts - so tickets correlate across channels. The shipped default template is byte-identical (pinned), unknown tokens stay literal, and a template that quotes a raw token is caught by the existing well-formedness check as a config error. Both SKUs, since the whole channel lives in the shared Notifications project.
 - **get_collection_health now carries a sweep_pressure verdict, so half-rate collection stops hiding behind 40 healthy collectors** ([#2296]) - two cross-region servers were collecting at half their configured cadence: their four heaviest collectors averaged ~60.7s of combined execution against a 60s sweep, so the serial collection body could never finish inside its interval, every relaunch was skipped (~50 service-log warnings/hour), and NOTHING else surfaced it - every collector reported HEALTHY, because from each one's own seat nothing was wrong. The tool now rolls the collectors' combined demand (average duration amortized by each collector's own cadence) against the minute the fastest cadence holds and serves busy_ms_per_minute / busy_percent / a verdict (OK, AT_RISK at 75%, SATURATED at 100%) plus the three heaviest contributors - attribution, because "which collectors spend the budget" is the actionable half of the answer. Deliberately built from the collectors' own execution times rather than delivered-gap statistics: at fleet scale the delivered cadence stretches benignly from bounded sweep concurrency (queueing), so gap-based detection would flag every server and drown the two that matter; execution demand is the arithmetic behind the watchdog's own "has not completed after Ns of EXECUTION" line and queueing cannot inflate it. The decision lives in the shared SweepPressureClassifier (PerformanceMonitor.Common) with the same decision table pinned in both suites, and both SKUs' tools serve the identical shape. Root-cause options for the two saturated servers (move them in-region, or lengthen their cadence) stay tracked on the issue - this change makes the condition visible either way.
 
@@ -2791,6 +2792,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#2246]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2246
 [#2300]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2300
 [#2312]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2312
+[#2320]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2320
 [#2306]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2306
 [#2302]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2302
 [#2296]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/2296

Darling/Darling.Tests/CpuAttributionTests.cs

@@ -0,0 +1,109 @@
+/*
+ * Copyright (c) 2026 Erik Darling, Darling Data LLC
+ *
+ * This file is part of the SQL Server Performance Monitor.
+ *
+ * Licensed under the MIT License. See LICENSE file in the project root for full license information.
+ */
+
+using PerformanceMonitor.Common;
+using Xunit;
+
+namespace Darling.Tests;
+
+/// <summary>
+/// Decision-table pins for the shared <see cref="CpuAttribution"/> (#2320) — the denominator the
+/// top-CPU rankings never handed the caller. This SAME table is pinned identically in Lite.Tests so
+/// the two SKUs cannot drift. The load-bearing cases are #2235's field numbers: the pre-fix reads
+/// explained ~10% of one box and nothing said so, and the Datadog disagreement died the moment its
+/// worker_time sum was divided by the box's consumed CPU-seconds and produced 137%.
+/// </summary>
+public sealed class CpuAttributionTests
+{
+    /* The #2235 window: 8 vCPU, 2 hours, RDS CPU averaging 18% → 10,368 measured core-seconds. */
+    private const double FieldAvgPct = 18.0;
+    private const int FieldCores = 8;
+    private const double FieldHours = 2.0;
+    private const int CoveredSamples = 120;
+    private const double FullSpan = 2.0;
+
+    [Fact]
+    public void TheFieldWindowComputesTheMeasuredDenominator()
+    {
+        var window = CpuAttribution.Compute(
+            attributedCpuMs: 3_000_000, FieldAvgPct, CoveredSamples, FullSpan, FieldCores, FieldHours);
+
+        Assert.NotNull(window);
+        Assert.Equal(10_368, window.Value.MeasuredSqlCpuSeconds, precision: 0);
+        Assert.Equal(3_000, window.Value.AttributedCpuSeconds, precision: 0);
+        Assert.Equal(0.289, window.Value.AttributedRatio, precision: 3);
+    }
+
+    /// <summary>Under half explained → the note says where the rest lives.</summary>
+    [Fact]
+    public void ALowRatioCarriesTheRemainderNote()
+    {
+        var window = CpuAttribution.Compute(3_000_000, FieldAvgPct, CoveredSamples, FullSpan, FieldCores, FieldHours);
+
+        Assert.NotNull(window!.Value.Note);
+        Assert.Contains("below the ranking cut", window.Value.Note, System.StringComparison.Ordinal);
+    }
+
+    /// <summary>
+    /// The impossibility detector: attributed exceeding measured by more than sampling skew could
+    /// explain gets the skew note — the exact division that settled #2235's Datadog claim (137%).
+    /// </summary>
+    [Fact]
+    public void AttributedBeyondMeasuredCarriesTheSkewNote()
+    {
+        var window = CpuAttribution.Compute(14_229_000, FieldAvgPct, CoveredSamples, FullSpan, FieldCores, FieldHours);
+
+        Assert.NotNull(window);
+        Assert.True(window.Value.AttributedRatio > 1.3);
+        Assert.Contains("exceeds the measured total", window.Value.Note, System.StringComparison.Ordinal);
+    }
+
+    /// <summary>An ordinary healthy ratio says nothing — notes are for the two failure directions.</summary>
+    [Fact]
+    public void AMidRatioCarriesNoNote()
+    {
+        var window = CpuAttribution.Compute(8_000_000, FieldAvgPct, CoveredSamples, FullSpan, FieldCores, FieldHours);
+
+        Assert.NotNull(window);
+        Assert.Null(window.Value.Note);
+    }
+
+    /// <summary>
+    /// Every way the denominator can be untrustworthy omits the window rather than fabricating one:
+    /// no average, no samples, unknown or nonsensical cores, a degenerate window, thin coverage,
+    /// and a hard-zero average (the ratio would divide by zero).
+    /// </summary>
+    [Fact]
+    public void AnUnsupportableDenominatorIsOmittedNeverFabricated()
+    {
+        Assert.Null(CpuAttribution.Compute(1_000, null, CoveredSamples, FullSpan, FieldCores, FieldHours));
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, 0, FullSpan, FieldCores, FieldHours));
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, CoveredSamples, FullSpan, null, FieldHours));
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, CoveredSamples, FullSpan, 0, FieldHours));
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, CoveredSamples, FullSpan, FieldCores, 0));
+        /* A 24h window whose samples span only 5 hours: the average extrapolates a fragment —
+           span-based on purpose, so a SLOW but steady cadence still qualifies (the review catch). */
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, 300, 5.0, FieldCores, 24));
+        /* Two lonely points can bracket a wide span — the sample floor rejects them. */
+        Assert.Null(CpuAttribution.Compute(1_000, FieldAvgPct, 2, FullSpan, FieldCores, FieldHours));
+        Assert.Null(CpuAttribution.Compute(1_000, 0.0, CoveredSamples, FullSpan, FieldCores, FieldHours));
+    }
+
+    /// <summary>
+    /// The span floor's boundary: samples spanning exactly half the window qualify — and cadence
+    /// never enters it, so a server whose cpu_utilization schedule was slowed to 5 minutes (24
+    /// samples in 2 hours) qualifies exactly like a 1-minute one (the review catch: a
+    /// count-per-minute expectation would have disqualified it permanently).
+    /// </summary>
+    [Fact]
+    public void SpanAtTheFloorQualifies_AtAnyCadence()
+    {
+        Assert.NotNull(CpuAttribution.Compute(1_000, FieldAvgPct, 60, FieldHours * 0.5, FieldCores, FieldHours));
+        Assert.NotNull(CpuAttribution.Compute(1_000, FieldAvgPct, 24, FullSpan, FieldCores, FieldHours));
+    }
+}

Darling/Darling.Tests/DarlingMcpDataToolsTests.cs

@@ -578,6 +578,11 @@ public async Task DataTools_ReadPlantedRows_AgainstDevPostgres()
             var newer = older.AddMinutes(1);
 
             await PlantCpuAsync(connection, newer, ct);
+            /* #2320: two more CPU samples, hours apart, so the attribution gate's floors are met for
+               the default 24h window (>= 3 samples spanning >= half of it) and cpu_window computes
+               through the REAL tool call — the wiring the pure-math pins cannot reach. */
+            await PlantCpuAsync(connection, newer.AddHours(-23), ct);
+            await PlantCpuAsync(connection, newer.AddHours(-13), ct);
             await PlantWaitStatsAsync(connection, older, newer, ct);
             await PlantMemoryStatsAsync(connection, newer, ct);
             await PlantMemoryClerksAsync(connection, newer, ct);
@@ -605,6 +610,19 @@ public async Task DataTools_ReadPlantedRows_AgainstDevPostgres()
             var q = await DarlingMcpDataTools.GetTopQueriesByCpu(postgres, ServerName);
             AssertServerEnvelope(q, "queries");
             Assert.Contains("0xE2EDATAHASH", q, StringComparison.Ordinal);   /* the planted query surfaced */
+
+            /* #2320: cpu_window computed END TO END — reader SQL against the real schema (a column
+               typo would hide behind the degrade-to-null catch forever, which is why this exists),
+               the planted 40% average x cpu_count 16 x the 24h window, and the tiny planted
+               numerator drawing the below-half note. */
+            using (var qDoc = JsonDocument.Parse(q))
+            {
+                var cpuWindow = qDoc.RootElement.GetProperty("cpu_window");
+                Assert.NotEqual(JsonValueKind.Null, cpuWindow.ValueKind);
+                Assert.Equal(0.40 * 16 * 24 * 3600, cpuWindow.GetProperty("measured_sql_cpu_seconds").GetDouble(), precision: 0);
+                Assert.True(cpuWindow.GetProperty("attributed_ratio").GetDouble() < 0.5);
+                Assert.Contains("below the ranking cut", cpuWindow.GetProperty("note").GetString(), StringComparison.Ordinal);
+            }
             AssertServerEnvelope(await DarlingMcpDataTools.GetTopProceduresByCpu(postgres, ServerName), "procedures");
             AssertServerEnvelope(await DarlingMcpDataTools.GetQueryStoreTop(postgres, ServerName), "queries");
 

Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs

@@ -421,6 +421,12 @@ public DarlingWorker(ILogger<DarlingWorker> logger, ILoggerFactory loggerFactory
         _mcpState = mcpState;
         _webState = webState;
         _registryState = registryState;
+
+        /* #2320 (round-4 review catch): set HERE, not only in the MCP host — mcp.enabled is OFF by
+           default, so on a default install the web dashboard's /api/read mirror is the only caller
+           of the top-CPU tools, and a logger set only by the MCP host would leave that path's
+           degrade-to-null silent. The worker always runs. */
+        Mcp.DarlingMcpDataTools.DegradeLogger = logger;
     }
 
     private sealed class ServerLoopState

Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs

@@ -155,6 +155,67 @@ FROM cpu_utilization_stats
         ORDER BY sample_time
         """;
 
+    /// <summary>
+    /// #2320: the attribution denominator's raw ingredients for one window — the average SQL-process
+    /// CPU percent and how many samples that average rests on (the coverage gate's input; the shared
+    /// CpuAttribution.Compute owns the trust decision). Windows on collection_time like every other
+    /// windowed read here; the de-skewed sample_time is irrelevant to an average.
+    /// </summary>
+    public static async Task<(double? AvgSqlCpuPercent, int Samples, double SpanHours)> GetCpuWindowAverageAsync(
+        NpgsqlDataSource postgres, int serverId, DateTime startUtc, DateTime endUtc, CancellationToken cancellationToken = default)
+    {
+        /* The span (last minus first sample) rides along because the coverage gate judges SPAN, not
+           count-per-minute — the collector's cadence is user-configurable, so a count expectation
+           would permanently disqualify a legitimately slowed server (the review catch). NULL-valued
+           rows are excluded up front (the round-9 catch): AVG would skip them anyway, but COUNT(*)
+           and the span must rest on the same rows the average does, or Samples overstates the
+           evidence behind it. */
+        const string sql = """
+            SELECT
+                AVG(sqlserver_cpu_utilization)::float8,
+                COUNT(*)::int,
+                COALESCE(EXTRACT(EPOCH FROM (MAX(collection_time) - MIN(collection_time))) / 3600.0, 0)::float8
+            FROM cpu_utilization_stats
+            WHERE server_id = $1
+            AND   collection_time >= $2
+            AND   collection_time <= $3
+            AND   sqlserver_cpu_utilization IS NOT NULL
+            """;
+        await using var command = postgres.CreateCommand(sql);
+        AddInt(command, serverId);
+        AddTimestamp(command, startUtc);
+        AddTimestamp(command, endUtc);
+        await using var reader = await command.ExecuteReaderAsync(cancellationToken);
+        if (!await reader.ReadAsync(cancellationToken))
+        {
+            return (null, 0, 0);
+        }
+
+        return (reader.IsDBNull(0) ? null : reader.GetDouble(0), reader.GetInt32(1), reader.IsDBNull(2) ? 0 : reader.GetDouble(2));
+    }
+
+    /// <summary>
+    /// #2320: the server's core count from its latest properties snapshot — the other half of the
+    /// denominator. Null when properties were never collected; the caller omits the ratio then.
+    /// </summary>
+    public static async Task<int?> GetLatestCpuCountAsync(
+        NpgsqlDataSource postgres, int serverId, CancellationToken cancellationToken = default)
+    {
+        /* COALESCE(vcore_count, cpu_count) — the same Azure-aware read the FinOps utilization CTE
+           uses in both SKUs: on Azure SQL DB the vcore count is the honest denominator. */
+        const string sql = """
+            SELECT COALESCE(vcore_count, cpu_count)
+            FROM server_properties
+            WHERE server_id = $1
+            ORDER BY collection_time DESC
+            LIMIT 1
+            """;
+        await using var command = postgres.CreateCommand(sql);
+        AddInt(command, serverId);
+        var result = await command.ExecuteScalarAsync(cancellationToken);
+        return result is int cores && cores > 0 ? cores : null;
+    }
+
     public static async Task<List<CpuSample>> GetCpuUtilizationAsync(
         NpgsqlDataSource postgres, int serverId, DateTime startUtc, CancellationToken cancellationToken = default)
     {