Repository navigation
Investigations: a re-fire joins a reported request quietly (#6778) - #6780
Merged
Merged
Conversation
erikdarlingdata
marked this pull request as ready for review
October 26, 2026 11:01
erikdarlingdata
marked this pull request as draft
October 26, 2026 11:37
erikdarlingdata
force-pushed
the
fix/6778-refire-joins-reported-request
branch
from
October 26, 2026 11:40
b7ff99d to
21a8cbd
Compare
erikdarlingdata
marked this pull request as ready for review
October 26, 2026 11:40
erikdarlingdata
marked this pull request as draft
October 26, 2026 12:01
An alert joins a reported request when its metric is already on it, its severity is not above the request's max_severity, the request has no resolved journal row, and it is inside the grouping window and the 4-hour cap. The join is journaled joined_quiet, which no reader delivers, and it counts against no limit. Darling and Lite get the same clause. Two older fixtures opened several reported requests on one server, family and metric and relied on each opening its own request. The follow-up test now gives each request its own metric. The export fixture now ends each case's grouping window 30 minutes after the time it says the case opened.
erikdarlingdata
force-pushed
the
fix/6778-refire-joins-reported-request
branch
from
October 26, 2026 12:02
21a8cbd to
c9c9d66
Compare
erikdarlingdata
marked this pull request as ready for review
October 26, 2026 12:02
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude posting for Erik Darling.
Problem
A reported investigation request is final (#5632). On the production store, one High CPU episode re-fired about every 5 minutes for two hours. The agent reported each request in about 12 seconds, so the episode made 22 requests with one alert each, and all 22 counted against the route's limits.
The change
FindJoinSqlin both stores now also finds areportedrequest when all of these hold: same server and family, the alert's metric is already on the request, the alert's severity rank is not abovemax_severity, and the request has noresolvedjournal row. The grouping window and the 4-hour cap apply as before. Theexpires_at > nowclause now applies to open and claimed rows only. The SQL takes the severity as a new parameter$6, andOpenOrJoinAsyncpasses it. TheOpenOrJoinAsyncremarks and the class summary state the new rule and why the throttle cannot decide it (the request is opened before the cooldown, for{{investigation_json}}, #5821).Behavior change
Identical: open and claimed requests join as before. A closed request stays final. A higher severity, a new metric, an alert after a resolve, an alert past the window or the 4-hour cap, and another server all open a new request. A join never counts against a limit. No schema change.
Changed: a qualifying alert on a reported request joins it. It is journaled
joined_quiet.JoinIsEventis already false for it (no new metric, no higher severity), so no reader delivers it,Eventedis false, and the request's state and report do not change. The dispatch callers only call the "opened" hook onOpenedor an evented join, so nothing treats the reported request as live (no claim, no brief rebuild, no event).Tests
Darling
InvestigationRequestStoreLiveTestsand the Lite twin: the old "reported is final" test now expects a quiet join. New cases: same metric and severity 5 minutes later (alert_count 2,joined_quiet, statereported, request count unchanged, window slides). Then: a higher severity, a new metric of the same family, another server (the first request alert_count stays 1), aresolvedrow, 31 minutes after the last alert, past the 4-hour cap, a closed request. No source-text tests.Two older fixtures opened several reported requests on one server, family and metric, and relied on each opening its own request. PR CI failed them on the first run.
InvestigationFollowUpLiveTestsnow gives each of its 21 requests its own metric.InvestigationFeedbackExportLiveTests.CaseAsyncnow ends each case's grouping window 30 minutes after the time it says the case opened, as a real case of that age would.What ran
InvestigationRequestStoreLiveTests,AgentActionGateTestsandExternalAlertDrainLiveTests, 105 tests, all pass exceptAUncOrFileLinkPlantedInTheStore_ReadsBack_AndRendersAsText. That one calls the WPF viewer, which I stubbed out in the scratch project because it cannot run on macOS. It is not run here.FindJoinSqlchange reverted, the quiet-join case, the claimed/reported case and the resolved/window/cap case fail. Restored, they pass.Considered, not added
resolvedjournal row. The row comes from a once-a-minute sweep and the engine re-evaluates about every 5 minutes, so a resolve and a re-fire are at least one evaluation apart.CHANGELOG entry
Investigations: an alert that keeps firing after a request is reported now joins that request quietly (same metric, same or lower severity, not resolved, inside the grouping window) instead of opening a new request each time. (#6778)
Refs #6778