[Muhammad Waqar Anwar (WaqarAnwar)]

Description

Closes #4036.

If the IN_PROGRESS update for a queued scenario run commits but the call still raises, the handoff retry sees a non-QUEUED row and drops the run. The row stays IN_PROGRESS with nothing executing it, and startup reconciliation only covers SQLiteMemory. This PR makes the retry start the run instead.

• Handoff retry: _handoff_scheduler_async now recognizes an IN_PROGRESS row whose saved start time matches the one this scheduler wrote for the run, and starts the run without writing to the database again. Other unexpected states are still dropped from the queue.
• Launch step: task creation moved into _launch_scheduled_run_locked so the retry can start a run without repeating the state update.
• Cancel during the retry window: cancelling a queued run now also matches IN_PROGRESS. Before, the update did nothing, the cancel raised ValueError, and the run had already left the queue, so it stayed stuck. A queued run has no task yet, so there is nothing else to stop.

The failure is simulated with a mocked database. It hasn't been observed in production or tried against a real Azure SQL connection failure.

[Copilot]

🟢 Approval recommended

No active review issues remain for the scheduler retry-recovery and cancellation changes.

No open issues 🎉

What changed in this PR

Fixes #4036 by allowing the scenario scheduler to start a queued run when its earlier IN_PROGRESS update committed but reported a failure.

Changes:

• Recognizes a persisted start timestamp belonging to the current scheduler and launches the run without repeating the database update.
• Extracts task launch logic and allows cancellation during the handoff retry window.
• Adds mocked regression tests for retry recovery, mismatched start timestamps, and cancellation.

File	Description
tests/​unit/​backend/​test_scenario_run_service.py	Adds regression coverage for handoff retry recovery and cancellation.
pyrit/​backend/​services/​scenario_run_service.py	Recovers committed starts on retry and handles cancellation before task launch.

_{Review effort: Balanced}

[Aman (its-amann)]

raw == isoformat() misses when metadata comes back as a datetime (or Z vs +00:00). _load_started_at already handles that:

         persisted_started_at: object = persisted.metadata.get(SCENARIO_RUN_STARTED_AT_METADATA_KEY)
         return (
             run.started_at is not None
             and persisted.scenario_run_state == ScenarioRunState.IN_PROGRESS
-            and persisted_started_at == run.started_at.isoformat()
+            and ScenarioRunService._load_started_at(scenario_result=persisted) == run.started_at
         )

[Muhammad Waqar Anwar (WaqarAnwar)]

Good catch, thanks. The value is written by us via .isoformat() and comes back from JSON metadata, so it should round-trip as the same string in practice. But comparing strings is fragile, and reusing _load_started_at is cleaner. Switched to comparing the parsed datetime and removed the unused variable. I also added a test where the stored timestamp uses Z instead of +00:00.