Skip to content

Investigations: a re-fire joins a reported request quietly (#6778) - #6780

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/6778-refire-joins-reported-request
Oct 26, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/6778-refire-joins-reported-request

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Claude posting for Erik Darling.

Problem

A reported investigation request is final (#5632). On the production store, one High CPU episode re-fired about every 5 minutes for two hours. The agent reported each request in about 12 seconds, so the episode made 22 requests with one alert each, and all 22 counted against the route's limits.

The change

FindJoinSql in both stores now also finds a reported request when all of these hold: same server and family, the alert's metric is already on the request, the alert's severity rank is not above max_severity, and the request has no resolved journal row. The grouping window and the 4-hour cap apply as before. The expires_at > now clause now applies to open and claimed rows only. The SQL takes the severity as a new parameter $6, and OpenOrJoinAsync passes it. The OpenOrJoinAsync remarks and the class summary state the new rule and why the throttle cannot decide it (the request is opened before the cooldown, for {{investigation_json}}, #5821).

Behavior change

Identical: open and claimed requests join as before. A closed request stays final. A higher severity, a new metric, an alert after a resolve, an alert past the window or the 4-hour cap, and another server all open a new request. A join never counts against a limit. No schema change.

Changed: a qualifying alert on a reported request joins it. It is journaled joined_quiet. JoinIsEvent is already false for it (no new metric, no higher severity), so no reader delivers it, Evented is false, and the request's state and report do not change. The dispatch callers only call the "opened" hook on Opened or an evented join, so nothing treats the reported request as live (no claim, no brief rebuild, no event).

Tests

Darling InvestigationRequestStoreLiveTests and the Lite twin: the old "reported is final" test now expects a quiet join. New cases: same metric and severity 5 minutes later (alert_count 2, joined_quiet, state reported, request count unchanged, window slides). Then: a higher severity, a new metric of the same family, another server (the first request alert_count stays 1), a resolved row, 31 minutes after the last alert, past the 4-hour cap, a closed request. No source-text tests.

Two older fixtures opened several reported requests on one server, family and metric, and relied on each opening its own request. PR CI failed them on the first run. InvestigationFollowUpLiveTests now gives each of its 21 requests its own metric. InvestigationFeedbackExportLiveTests.CaseAsync now ends each case's grouping window 30 minutes after the time it says the case opened, as a real case of that age would.

What ran

  • Both test projects build with 0 errors and 0 warnings.
  • Live PostgreSQL (timescaledb pg17, scratch project outside the repo): InvestigationRequestStoreLiveTests, AgentActionGateTests and ExternalAlertDrainLiveTests, 105 tests, all pass except AUncOrFileLinkPlantedInTheStore_ReadsBack_AndRendersAsText. That one calls the WPF viewer, which I stubbed out in the scratch project because it cannot run on macOS. It is not run here.
  • Mutation: with the FindJoinSql change reverted, the quiet-join case, the claimed/reported case and the resolved/window/cap case fail. Restored, they pass.
  • Lite tests are built only. CI runs them. The full suite was not run.

Considered, not added

  • Reading the resolution rows directly instead of the resolved journal row. The row comes from a once-a-minute sweep and the engine re-evaluates about every 5 minutes, so a resolve and a re-fire are at least one evaluation apart.

CHANGELOG entry

Investigations: an alert that keeps firing after a request is reported now joins that request quietly (same metric, same or lower severity, not resolved, inside the grouping window) instead of opening a new request each time. (#6778)

Refs #6778

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 26, 2026 11:01
@erikdarlingdata
erikdarlingdata marked this pull request as draft October 26, 2026 11:37
@erikdarlingdata
erikdarlingdata force-pushed the fix/6778-refire-joins-reported-request branch from b7ff99d to 21a8cbd Compare October 26, 2026 11:40
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 26, 2026 11:40
@erikdarlingdata
erikdarlingdata marked this pull request as draft October 26, 2026 12:01
An alert joins a reported request when its metric is already on it, its severity is not above the request's max_severity, the
request has no resolved journal row, and it is inside the grouping window and the 4-hour cap. The join is journaled joined_quiet,
which no reader delivers, and it counts against no limit. Darling and Lite get the same clause.

Two older fixtures opened several reported requests on one server, family and metric and relied on each opening its own request.
The follow-up test now gives each request its own metric. The export fixture now ends each case's grouping window 30 minutes after
the time it says the case opened.
@erikdarlingdata
erikdarlingdata force-pushed the fix/6778-refire-joins-reported-request branch from 21a8cbd to c9c9d66 Compare October 26, 2026 12:02
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 26, 2026 12:02
@erikdarlingdata
erikdarlingdata merged commit 20ae8a3 into dev Oct 26, 2026
30 of 38 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/6778-refire-joins-reported-request branch October 26, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant