Restore silently swallows sub-agent materialization/terminal-transition failures (bare catch, no logging) leaving reloaded sub-agents stuck Running with no diagnostic

[JoshuaRowePhantom]

Summary

During Copilot SDK session reload/restore, AgentChat.MarkRestoredSubAgentTerminalAsync materialises each persisted sub-agent (acquires its lease, which loads the child's transcript from the persistence store) and forces it to the terminal Succeeded state added by #1128. That whole operation is wrapped in a bare catch { } that silently discards every exception with no logging, no diagnostic, and no rethrow. If a restored sub-agent fails to materialise — a corrupt/undeserialisable persisted session, a store read failure, a factory error, or any exception from AcquireLeaseAsync — the failure is completely invisible: the sub-agent silently stays in its default (Running) state, the #1128 forced-Succeeded transition never happens (so the UI shows a perpetual pulsating-brain / running marker), and there is no log line, error diagnostic, or surfaced fault anywhere to indicate that a child transcript failed to load. Operators and developers get a silently-degraded restore with no signal.

This is a robustness / observability defect on the restore path, distinct from the persistence mechanism itself: the transcript-persistence and reload machinery is otherwise correct (see "Not a defect" note below), but its failure mode during restore is swallowed.

Root Cause

Phantom.Workspaces.Llm.Core\AgentChat.cs:1188-1210 — MarkRestoredSubAgentTerminalAsync:

private async Task MarkRestoredSubAgentTerminalAsync(
    SubAgent stub,
    CancellationToken cancellationToken)
{
    try
    {
        var lease = await stub.AcquireLeaseAsync(cancellationToken).ConfigureAwait(false);

        // Keep the lease alive for as long as this parent lives so the terminal
        // completionStateOverride persists across other lease acquisitions.
        this.RegisterOwnedResource(lease);

        lease.AgentChat.SetCompletionState(AgentChatCompletionState.Succeeded);
    }
    catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
    {
    }
    catch
    {
        // Best-effort: leaving the sub-agent in its default state is preferable to
        // faulting parent restore.
    }
}

• The try body wraps the child materialisation / transcript load (stub.AcquireLeaseAsync → IRunningAgentChatFactory.GetAsync → AgentChat.CreateAsync → InitializeAsync → RestoreAsync + ReadMessagesAsync + LoadInitialHistory) as well as lease registration and the Reloading a Copilot SDK session should mark all restored SDK sub-agents completed #1128 SetCompletionState(Succeeded) call.
• The first catch correctly and narrowly handles cooperative cancellation.
• The second catch is bare (line 1205-1209): it catches all exception types, contains no ILogger/Console/Trace call, does not rethrow, and does not surface any diagnostic. It swallows the failure entirely.

RestoreSubAgentsAsync (AgentChat.cs:1121-1168) invokes this as fire-and-forget work (this.restoredSubAgentTerminalTasks.Add(terminalTask), AgentChat.cs:1162-1166), so even a faulted Task is never observed except by the test-only WaitForRestoredSubAgentsMarkedTerminalAsync (AgentChat.cs:1178-1186). In production nothing awaits or inspects these tasks, compounding the silent-swallow: a materialisation failure produces neither an exception, a log, nor a faulted-task observation.

The design intent ("leaving the sub-agent in its default state is preferable to faulting parent restore") is reasonable — a single bad child should not abort the whole parent restore — but swallowing without any observability is the defect. The correct behaviour is to keep restore resilient and surface the failure (log it, and/or emit a diagnostic on the affected sub-agent), so that a silently-empty / silently-stuck-Running sub-agent is diagnosable rather than invisible.

Not a defect (verified during diagnosis)

A prior hypothesis claimed completed hosted sub-agent transcripts are never persisted under a reloadable child record. This was verified false. Every AgentChat, including a hosted sub-agent, is given the parent's ConfiguredStore (AgentChatFactory.cs:155-164) and wraps its resolved client in a StreamingPersistenceMiddleware over an IncrementalPersistenceChatHistoryProvider (AgentChat.cs:277-281) that writes the child's AgentSessionJson + NewMessages to the store under the child's own AgentSessionId (IncrementalPersistenceChatHistoryProvider.cs:135-147). The hosted run streams through that middleware (RunHostedProcessLoopAsync → StartRun → session.RunStreamAsync over ChatClientAgent(streamingMiddleware, …), AgentChat.cs:1815-1819, 2003-2004, 304). On restore, InitializeAsync reloads it via RestoreAsync + ReadMessagesAsync + LoadInitialHistory (AgentChat.cs:342-345, 376), and the existing test AgentChatPersistenceTests.InitializeAsync_RestoredSubAgent_ChatHistoryLoaded asserts the restored child History.Count > 0. So transcript persistence/reload itself works; only its failure path during restore is swallowed, which is what this issue addresses.

Affected Files

File	Contribution
Phantom.Workspaces.Llm.Core\AgentChat.cs	MarkRestoredSubAgentTerminalAsync (1188-1210) — bare catch (1205-1209) silently swallows all child-materialisation / transcript-load / terminal-transition failures with no logging or diagnostic. RestoreSubAgentsAsync (1121-1168) schedules it fire-and-forget (1162-1166); the resulting Tasks are only observed by the test-only WaitForRestoredSubAgentsMarkedTerminalAsync (1178-1186), never in production.
Phantom.Workspaces.Llm.Core\SubAgent.cs	AcquireLeaseAsync materialises the child (factory.GetAsync), the operation whose exceptions are swallowed; a materialisation failure leaves the stub un-materialised and stuck in the default Running state.

Design / Fix

Goal: keep restore resilient (a single failing sub-agent must not fault the whole parent restore, preserving #1128's per-child best-effort behaviour) while making the failure observable instead of silently discarded.

1 — Replace the bare catch with an observing catch. Capture the exception and surface it. AgentChat does not currently hold an ILogger, but it already has a first-class user-visible diagnostic channel used throughout the class — AddHistoryItem / EnqueueSystemNote with AgentChatHistoryItem.DiagnosticChatRole and ErrorContent (e.g. AgentChat.cs:698, 840, and the hosted-loop error diagnostic at 1840-1856). Emit a diagnostic (on the parent, identifying the child by session id) and/or a log so the failure is visible:

catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
}
catch (Exception ex)
{
    // Best-effort: do not fault parent restore for one bad child, but the failure
    // MUST be observable (issue #<THIS>). Surface a diagnostic identifying the child.
    this.AddHistoryItem(new AgentChatHistoryItem
    {
        Role = AgentChatHistoryItem.DiagnosticChatRole,
        Contents = [new ErrorContent(
            $"Failed to restore sub-agent '{stub.AgentSessionId}': {ex.Message}")],
        Timestamp = this.timeProvider.GetUtcNow(),
    });
    // and/or ILogger.LogError(ex, ...) if a logger is threaded into AgentChat.
}

2 — Do not leave a failed restored sub-agent silently Running. Because the whole point of #1128 is that a reloaded sub-agent has no live SDK run to ever move it out of the default Running state, a materialisation failure that skips SetCompletionState(Succeeded) leaves the UI showing a perpetual running marker. On the failure path, still drive the stub to a terminal state so the running indicator clears (e.g. mark it Failed via the stub, or force Succeeded on a best-effort re-attempt), consistent with #1128's requirement that restored sub-agents resolve to a terminal state.

3 — Observe the fire-and-forget tasks. RestoreSubAgentsAsync stores each terminalTask only for the test helper; in production a faulted task is never observed. Attach a continuation (or await within the session-init flow) that logs/surfaces faults so a swallowed-then-faulted task cannot disappear.

Reconciliation with #1128 / #1140 / #1139

• Reloading a Copilot SDK session should mark all restored SDK sub-agents completed #1128 (restored SDK sub-agents forced to Succeeded): this fix preserves that behaviour on the success path and extends it to the failure path (still resolve to a terminal state so indicators clear), while making failures observable. The try body being fixed is exactly the Reloading a Copilot SDK session should mark all restored SDK sub-agents completed #1128 transition.
• Reloading a session bumps completed sub-agents' last-updated time to the reload time (should preserve original) #1140 (restore must preserve the original last-updated timestamp, not stamp reload time): compatible — the diagnostic added here is emitted only on failure and does not touch lastUpdatedAt; the success path is unchanged, so Reloading a session bumps completed sub-agents' last-updated time to the reload time (should preserve original) #1140's preserveLastUpdatedAt behaviour is unaffected.
• Currently-running sub-agent's tool calls + assistant text are dropped — rendered in neither the sub-agent nor the parent transcript #1139 (currently-running sub-agent shows no live transcript — a live routing/attribution bug): distinct and unrelated to restore. Currently-running sub-agent's tool calls + assistant text are dropped — rendered in neither the sub-agent nor the parent transcript #1139 is about a live SDK run misattributing untagged content to the parent; this issue is about the reload path's silent failure handling. Cross-linked as related-area only.

Expected Tests

Reuse existing classes/style (xUnit [Fact], Subject_Scenario_ExpectedOutcome) from AgentChatResumeTests, AgentChatHostedSubAgentTests, and AgentChatPersistenceTests in Phantom.Workspaces.Llm.Core.Tests (helpers: InMemoryAgentPersistenceStore, DeterministicTestChatClient, CapturingTaskScheduler, AgentChatFactory, WaitForRestoredSubAgentsMarkedTerminalAsync).

Test Name	Class	What It Verifies
AgentChat_Resume_SubAgentMaterializationFails_SurfacesDiagnostic_NotSwallowed	AgentChatResumeTests	When AcquireLeaseAsync throws during restore (e.g. store/deserialisation failure), the failure is surfaced as a DiagnosticChatRole/ErrorContent history item (or logged), not silently discarded.
AgentChat_Resume_SubAgentMaterializationFails_DoesNotFaultParentRestore	AgentChatResumeTests	A single failing restored sub-agent does not fault the parent restore; the parent completes initialisation and other sub-agents still restore (preserves the best-effort intent).
AgentChat_Resume_SubAgentMaterializationFails_StillResolvesToTerminalState	AgentChatResumeTests	A restored sub-agent whose materialisation fails is still driven to a terminal state so the UI running indicator clears rather than showing a perpetual Running (reconciles with #1128).
RestoreSubAgentsAsync_SubAgentMaterializationFails_TerminalTaskObservedNotDropped	AgentChatHostedSubAgentTests	The fire-and-forget terminal task's fault is observed/surfaced (via continuation or session-init flow) and not left as an unobserved faulted Task.
AgentChat_HostedSubAgent_PersistThenReload_RestoresAssistantTextAndToolCalls	AgentChatHostedSubAgentTests	(closes the round-trip coverage gap) A hosted sub-agent that streamed assistant text and tool calls through the SDK/persistence middleware, when persisted and then reloaded via restore, exposes both the original assistant text and the tool-call items in the restored child's History — asserting content, not just History.Count > 0.

[JoshuaRowePhantom]

Related restore-area issues: #1128 (forces restored SDK sub-agents to Succeeded — this issue makes the failure path of that transition observable), #1140 (restore must preserve original last-updated timestamp — compatible; the diagnostic here is failure-only and does not touch lastUpdatedAt), and #1139 (distinct: live running-sub-agent routing/attribution, not the reload path). Diagnosis also verified that hosted sub-agent transcripts ARE persisted under a reloadable child AgentSessionId and DO reload (AgentChatPersistenceTests.InitializeAsync_RestoredSubAgent_ChatHistoryLoaded), so the earlier 'transcripts never persisted' hypothesis was rejected; only the restore failure handling is defective.