046_RepairAutomaticSettlementTimestamps misses pre-fix auto-settle events tagged actor_kind: 'client'

[HelgiSteinarr-mila]

What happened

After updating, recent threads that used Codex appeared to be "gone," and threads using Codex that the user didn't recognize as recent appeared in their place. Separately, almost all threads — including ones months old — were labeled as settled "~4 days old," which was not correct.

Diagnosis

Nothing was actually deleted: every thread with a thread.created event in orchestration_events was still present in projection_threads (58/58). The real problem is the settled_at timestamp used by the sidebar to sort and label the Settled section (resolveSettledThreadTimestamp in packages/client-runtime/src/state/threadSort.ts, which returns settledAt when present).

During an earlier update (before this user's most recent one), an automatic settlement sweep ran a burst of 34+ thread.settled commands roughly one per second over ~90 seconds, settling nearly every idle thread across every project. Each event carries actor_kind: 'client' and a plain random-UUID command_id — not the server:auto-settle:<threadId>:<uuid> / actor_kind: 'server' shape the current ThreadSettlementReactor uses. Each event's payload_json.settledAt equals its own occurred_at (the sweep time), not the thread's real last activity — the exact bug described in the comment on apps/server/src/persistence/Migrations/046_RepairAutomaticSettlementTimestamps.ts ("Server auto-settlement used to stamp settledAt with the sweep time instead of the thread's last activity").

Migration 046 exists specifically to repair this, but its detection query is scoped to:

WHERE aggregate_kind = 'thread'
  AND event_type = 'thread.settled'
  AND actor_kind = 'server'
  AND command_id LIKE 'server:auto-settle:%'
  AND json_extract(payload_json, '$.settledAt') = occurred_at

Because the pre-fix implementation apparently attributed these commands to actor_kind: 'client' with a plain UUID command_id (rather than the current server:auto-settle: shape), migration 046 silently skips every one of them. On this machine, comparing each burst-settled thread's settled_at against its true last activity (max of projection_thread_messages.created_at / projection_turns.requested_at|started_at|completed_at) found 42 threads misdated, several by months (e.g. a thread last active 2026-04-14 showed settled_at = 2026-09-04).

Net user-visible effect: the Settled section's "X days ago" labels cluster within a few days of the sweep regardless of a thread's true age, burying recently-relevant threads under old ones that now look deceptively recent, and making genuinely old, unfamiliar threads (e.g. ones run from a non-project directory like ~/Downloads) look like new/foreign activity.

Steps to reproduce

1. Be on a version with the pre-fix client-attributed auto-settle path (before ThreadSettlementReactor / server:auto-settle: commands existed).
2. Let the automatic inactivity sweep fire across a backlog of idle threads. Each resulting thread.settled event has actor_kind: 'client', a random command_id, and payload_json.settledAt == occurred_at.
3. Update to a version containing migration 046.
4. Run the migration and inspect projection_threads.settled_at for threads settled in step 2 — the value still equals the sweep time, not the thread's real last activity.

Version

Server 0.0.40 (tag v0.0.40, commit 09e8de9). Bad data originated on desktop app 0.0.35–0.0.38 per event metadata_json.origin.appVersion.

Environment

macOS 24.6.0 (Darwin), Apple silicon, Node v26.8.1. Desktop app, local server (http://127.0.0.1:3773). Providers in use: codex, claudeAgent.

Evidence

-- Detection query mirroring 046 but for actor_kind='client':
SELECT stream_id, occurred_at,
       json_extract(payload_json,'$.settledAt') AS settled_at
FROM orchestration_events
WHERE aggregate_kind='thread' AND event_type='thread.settled' AND actor_kind='client'
  AND json_type(payload_json,'$.settledAt')='text'
  AND json_extract(payload_json,'$.settledAt') = occurred_at
  AND occurred_at BETWEEN '2026-09-03T00:00:00.000Z' AND '2026-09-04T23:59:59.999Z';
-- 42 rows, e.g.:
--   thread "Backend billing report endpoint": real last activity 2026-04-14T13:21:02Z,
--     settled_at written as 2026-09-04T13:33:16Z (burst time)
--   thread "Port service interface linkage": real last activity 2026-07-01T14:56:47Z,
--     settled_at written as 2026-09-04T13:33:49Z (burst time)

Related issues

None found matching this specific migration gap. #9600 ("Old auto-settled threads re-appeared as unsettled in sidebar after version upgrade", closed) is a related but distinct symptom from around the same time window. Not a duplicate of #10933 (agent-session import namespace collision) — no import:-prefixed threads exist on this machine; every affected thread here is native.

Fix applied or workaround

With the user's explicit permission, ran a corrected version of migration 046's repair query against their local projection_threads, scoped to actor_kind = 'client' events with settledAt == occurred_at in the affected time window, recomputing settled_at from each thread's real last message/turn activity (falling back to created_at). 41 of 42 candidate rows were updated (one already matched). No source files were changed; no orchestration_events rows were modified — only the projection_threads.settled_at projection.

Suggested upstream fix: broaden migration 046's detection (or add a follow-up migration) to also match actor_kind = 'client' settlement events whose payload_json.settledAt equals occurred_at, excluding genuine one-off manual archives if possible (e.g. by requiring the event be part of a tight burst of many such events, or by shipping a fixed cutover date for when the server-side reactor replaced the older path).

Filed by

claude (opus-5) via t3 triage

[juliusmarminge]

Triage

Confirmed on current main. Migration 046 only repairs server-attributed server:auto-settle:% events. Client-attributed thread.settled rows with settledAt == occurred_at are skipped on purpose (046’s tests treat them as manual). That leaves a hole for the pre-reactor / client-dispatched auto-settle burst in this report.

What the code does

inferActorKind in OrchestrationEventStore labels a command client unless command_id starts with server: or provider:. Since #8600, ThreadSettlementReactor always dispatches thread.auto-settle with server:auto-settle:${threadId}:${uuid}, so those events are actor_kind = 'server'.

The decider still stamps a fresh thread.settle (the client command) as settledAt = occurredAt. thread.auto-settle uses the command’s settledAt (last activity since #9254).

046’s automatic_settlements CTE requires all of:

• actor_kind = 'server'
• command_id LIKE 'server:auto-settle:%'
• json_extract(payload_json, '$.settledAt') = occurred_at

The 046 test helper maps any non-server: command id to client and leaves that row unrepaired (thread-manual). That matches this machine: 42 client events with settledAt == occurred_at in a ~90s burst, several months off last activity.

resolveSettledThreadTimestamp returns settledAt when present, so the Settled shelf’s “X days ago” labels and order all collapse to the sweep.

047–049 do not touch settled_at. No open PR widens 046. This user is already on 0.0.40 (09e8de9), so 046 has already run — changing 046 in place will not re-run.

Going forward the reactor is fine. This is historical projection repair only. Event payloads can stay as recorded (same rationale as 046: engine/projectors bootstrap from projection rows and cursors).

Related

• [Bug]: Auto-settlement marks old threads fresh and sends completion pushes on first launch #9057 (closed) / fix(server): preserve automatic settlement timestamps #9254 (merged) — same wrong-stamp symptom for the server:auto-settle path. This is the leftover client-attributed cohort, not a reopen.
• fix(server): stop stale settlement notifications #9081 (open) — suppresses stale settlement notifications. Does not repair settled_at.
• [Bug]: Old auto-settled threads re-appeared as unsettled in sidebar after version upgrade #9600 (closed) / fix(server): settle inactive threads with open PRs #9610 (merged) — open-PR inactivity guard. Different symptom.
• Imported agent sessions duplicate T3-native threads: import dedup never consults the native thread namespace #10933 (open) — import-namespace collision. Reporter has no import: threads; not a duplicate.

Suggested fix

Add a follow-up migration (next id is 050). Do not edit 046.

Reuse 046’s projection-only update: set projection_threads.settled_at to the latest message/turn activity at or before the event’s occurred_at, else created_at. Leave orchestration_events alone. Keep the “projection settled_at still equals the bad event stamp” guard so a later manual re-settle is not rewritten.

Do not match every actor_kind = 'client' AND settledAt = occurred_at row. That is also a genuine one-off thread.settle today. Discriminate automated sweeps, e.g.:

• a tight cluster of many such client thread.settled events (this report: 34+ in ~90s), and/or
• a bounded cutover (desktop 0.0.35–0.0.38 / around the fix(server): settle threads server-side #8600 window, e.g. through ~2026-09-04), and
• optionally a large gap between settled_at and last activity.

Reporter already ran that repair locally with permission (41/42 rows; events unchanged).

Labels: bug, accepted, via-triage