test(evals): assert executor model fallback on primary failure

Add executor-falls-back-to-secondary-model: the primary call rejects, the
Agent block has a fallback model, and the handler serves the answer from
gpt-4o-mini. Asserts providerCalls === 2 and lastRequestModel, and fails
without the fallback row (checked locally: got gpt-4o, run errored).

Author: sudoKrishna

apps/sim/evals/README.md

@@ -119,8 +119,10 @@ Add a case to `EXECUTOR_SCENARIOS` in `executor-harness.ts`:
 - `expect` uses the loop's checks plus `resolvedInput` (a substring that must
   reach the provider messages), `succeeds` (expected `ExecutionResult.success`),
   and `providerCalls` (exact provider call count).
-- Set `agent.retry` to exercise the executor's per-block retry policy; make the
-  first `providerResponse` a `reject` and the retry lands on the next one.
+- Set `agent.retry` to exercise the executor's per-block retry policy, or
+  `agent.fallbackModels` to exercise model fallback. Make the first
+  `providerResponse` a `reject` and the next one succeeds; assert
+  `providerCalls` and `lastRequestModel` to prove which path recovered.
 
 Both suites write one report, so executor rows appear alongside loop rows.
 
@@ -135,8 +137,7 @@ model/tool time, first-response time, and token usage.
 ## Scope and next steps
 
 Two harnesses share one result shape and report: the tool loop and the
-`DAGExecutor`. The executor suite covers block retry —
-`executor-retries-failed-block` makes the first provider call reject, the block
-is replayed, and the run completes. Model fallback (`fallbackModels`) is not
-asserted yet. Further expansion (context/memory, model routing, subagent
-orchestration) is tracked as follow-up work.
+`DAGExecutor`. The executor suite covers both recovery paths — block retry
+(`executor-retries-failed-block`) and model fallback
+(`executor-falls-back-to-secondary-model`). Further expansion (context/memory,
+model routing, subagent orchestration) is tracked as follow-up work.

apps/sim/evals/agent-tool-use/executor-harness.ts

@@ -56,6 +56,8 @@ export interface ExecutorScenario {
     temperature?: number
     /** Enables the executor's per-block retry policy for the Agent block. */
     retry?: { enabled: boolean; maxTries: number; waitBetweenTriesMs: number }
+    /** Ordered models the Agent handler tries after the primary fails. */
+    fallbackModels?: Array<{ model: string }>
   }
   /** One entry per model call; the last entry serves any extra/retry calls. */
   providerResponse: ExecutorProviderResponse | ExecutorProviderResponse[]
@@ -66,6 +68,8 @@ export interface ExecutorScenario {
     succeeds?: boolean
     /** Exact number of provider calls the executor made. */
     providerCalls?: number
+    /** Model id sent on the final provider call (proves which candidate served). */
+    lastRequestModel?: string
   }
 }
 
@@ -90,6 +94,7 @@ function buildWorkflow(scenario: ExecutorScenario): SerializedWorkflow {
     ...(scenario.agent.temperature !== undefined
       ? { temperature: scenario.agent.temperature }
       : {}),
+    ...(scenario.agent.fallbackModels ? { fallbackModels: scenario.agent.fallbackModels } : {}),
   }
   if (scenario.agent.retry) agent.retry = scenario.agent.retry
 
@@ -198,6 +203,15 @@ export async function runExecutorScenario(
     })
   }
 
+  if (scenario.expect.lastRequestModel !== undefined) {
+    const lastModel = (requests.at(-1) as { model?: string } | undefined)?.model
+    checks.push({
+      name: 'last-request-model',
+      passed: lastModel === scenario.expect.lastRequestModel,
+      detail: `expected ${scenario.expect.lastRequestModel}, got ${String(lastModel)}`,
+    })
+  }
+
   const tokens = (output.tokens ?? {}) as { input?: number; output?: number; total?: number }
 
   return {
@@ -307,4 +321,31 @@ export const EXECUTOR_SCENARIOS: ExecutorScenario[] = [
       providerCalls: 2,
     },
   },
+  {
+    id: 'executor-falls-back-to-secondary-model',
+    name: 'falls back to the secondary model when the primary fails',
+    category: 'recovery',
+    description:
+      'The primary model call rejects and the Agent block has a fallback model. The handler must serve the answer from the fallback and the run must complete.',
+    workflowInput: { message: 'What is the API rate limit?' },
+    agent: {
+      model: 'gpt-4o',
+      userPrompt: 'What is the API rate limit?',
+      fallbackModels: [{ model: 'gpt-4o-mini' }],
+    },
+    providerResponse: [
+      { reject: '429 rate limited', content: '' },
+      {
+        content: 'The API rate limit is 100 requests per minute.',
+        model: 'gpt-4o-mini',
+        tokens: { input: 10, output: 20, total: 30 },
+      },
+    ],
+    expect: {
+      succeeds: true,
+      finalContent: '100 requests per minute',
+      providerCalls: 2,
+      lastRequestModel: 'gpt-4o-mini',
+    },
+  },
 ]