Python: fix(core): keep finished background tasks until cleared or released

[Karthik Thota (karthik-0306)]

Motivation & Context

1. Why is this change required?
   In the single-process BackgroundAgentsProvider, session state is persisted by the host only after an agent run completes. If the host restores a session snapshot taken while a task was RUNNING (e.g. after a run failure, turn rollback, or host state reload), any completed or failed task that had already been finalized in memory had its asyncio.Task purged from _RuntimeState.in_flight_tasks. On subsequent turn invocations, _refresh_task_state could not find the task in in_flight_tasks and misclassified it as LOST, discarding a result or error that had already completed successfully.

2. What problem does it solve?
   It ensures that completed/failed task instances remain available in memory so that a stale persisted RUNNING record can reconcile against the finished task and return the actual result or exception rather than reporting "Task state was lost".

3. What scenario does it contribute to?
   Single-process background agents executing across multi-turn sessions where host persistence timing, rollbacks, or intermediate turn failures could otherwise orphan completed tasks.

4. Issue Link:
   Part of [Feature]: BackgroundAgentsProvider cannot tell a task that is running on another process from one whose owner died #8760 (Track 1 — single-process in-flight task retention).

---

Description & Review Guide

• What are the major changes?

  • _background_agents.py: Removed runtime.in_flight_tasks.pop(task_info.id, None) from _finalize_task. Tasks are now retained in in_flight_tasks until explicitly removed via background_agents_clear_completed_task or session teardown via BackgroundAgentsProvider.release_session.
  • Added docstring and inline comments explaining the lifecycle rationale and retention policy.
  • test_harness_background_agents.py: Added 5 deterministic regression tests (using await task instead of asyncio.sleep):
    1. Pre-finalization snapshot restoration returns the real result instead of "Task state was lost".
    2. Failed task error text is preserved upon snapshot restore.
    3. Cancelled task status and cancellation error text are preserved upon snapshot restore.
    4. clear_completed_task explicitly removes the task from runtime.in_flight_tasks.
    5. release_session clears all in-flight tasks and releases the runtime state.

• What is the impact of these changes?

  • Eliminates false LOST task states caused by session persistence desync in single-process scenarios.
  • Memory trade-off: Completed asyncio.Task references remain in memory for the duration of the session until cleared by the caller or released with the session.

• What do you want reviewers to focus on?

  • The retention lifecycle in _finalize_task vs clear_completed_task / release_session.
  • Determinism and assertions in the newly added regression tests in test_harness_background_agents.py.

---

Validation

• pytest: 52 passed, 0 failed in 0.77s
• ruff check: All checks passed
• ruff format: Verified clean
• pyright: 0 errors, 0 warnings, 0 informations
• mypy: Success: no issues found in 1 source file

---

Related Issue

Part of #8760 (Does not close #8760; addresses Track 1 single-process gap while Track 2 cross-process ADR remains open).

---

AI Assistance

• No material AI assistance was used.
• This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details:
AI assisted with drafting regression tests, formatting docstrings, running validation checks (pytest, ruff, pyright, mypy), and drafting this PR description.

---

Contribution Checklist

• The code builds clean without any errors or warnings
• All unit tests pass, and I have added new tests where possible
• The PR follows the Contribution Guidelines
• This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
• This is not a breaking change.

[Copilot]

🟡 Changes recommended

Reused task IDs can associate stale snapshots with results from a later continuation.

1 open finding

• Retained task ID can apply results to the wrong execution · New

What changed in this PR

Retains finished background tasks to reconcile stale single-process session snapshots.

Changes:

• Stops removing tasks during finalization.
• Adds regression and lifecycle cleanup tests.

File	Description
_background_agents.py	Retains finalized tasks and documents cleanup lifecycle.
test_harness_background_agents.py	Tests snapshot recovery, failures, cancellation, clearing, and release.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[Eduard van Valkenburg (eavanvalkenburg)]

Karthik Thota (@karthik-0306) background_agents_clear_completed_task can clear a newer running continuation when the caller restores an older completed generation of the same task id. The restored generation passes the non-running check, but runtime.clear_task(task_id) removes every (task_id, generation) entry and the shared background session. A newer generation can therefore keep running untracked; later refreshes mark it lost, and session release can no longer cancel or drain it.

Please make clear generation-aware so it removes only the completed generation, or explicitly reject/cancel newer running generations before removing their runtime handles. Add a regression that restores generation 1 while generation 2 is still running and then attempts to clear generation 1.

[Karthik Thota (karthik-0306)]

Karthik Thota (Karthik Thota (@karthik-0306)) background_agents_clear_completed_task can clear a newer running continuation when the caller restores an older completed generation of the same task id. The restored generation passes the non-running check, but runtime.clear_task(task_id) removes every (task_id, generation) entry and the shared background session. A newer generation can therefore keep running untracked; later refreshes mark it lost, and session release can no longer cancel or drain it.

Please make clear generation-aware so it removes only the completed generation, or explicitly reject/cancel newer running generations before removing their runtime handles. Add a regression that restores generation 1 while generation 2 is still running and then attempts to clear generation 1.

Addressed in 6b90a3d. Generation tracking is removed. background_agents_clear_completed_task now drops the runtime handle and background session only when the local task is done or absent. If a continuation is still running, its handle stays tracked, so release_session(cancel_running=True) can still cancel and drain it. The persisted record is still removed because the caller asked to clear it. Regression added: restore the completed snapshot while the continuation runs, clear, then confirm the continuation is still tracked and release cancels it.

[Eduard van Valkenburg (eavanvalkenburg)]

Karthik Thota (@karthik-0306) The final head again keys retained runtime tasks only by task_id. background_agents_continue_task() overwrites that entry for a newer execution, while _refresh_task_state() reconciles any persisted RUNNING record against whichever task currently occupies the ID. A restored stale snapshot can therefore receive the result or error of a later continuation. Please preserve a serialized execution identity/generation (or otherwise prove the retained task is the same execution) and add a stale-snapshot-plus-continuation regression.

[Karthik Thota (karthik-0306)]

Eduard van Valkenburg (@eavanvalkenburg) Restored an execution identity. BackgroundTaskInfo now persists a generation (defaults to 1 for older state), and it goes up on every continuation. Runtime records which generation each tracked run belongs to, and refresh, wait and finalize only match a record's own generation. So a restored stale RUNNING snapshot becomes LOST instead of receiving a later continuation's result or error. Continuing from a stale snapshot is rejected while a newer run is still going, and otherwise moves past every generation seen. Clearing only releases runtime state when no newer generation exists. Added regression tests: restore a gen-1 RUNNING snapshot after gen 2 completes or fails, stale wait, and stale continue/clear.