toolUseBehavior: run_llm_again (default) doesn't work

[bakikucukcakiroglu]

Even if toolUseBehavior of an agent is set as run_llm_again (it is the default behavior but I still set it to be sure), agent is not called again after tool calls. Instead the agent run that asks for tool calls also creates an output, and it is given as the final output of the multi agent system. The tools are also called later on.

It was not an issue (it was too rare) while I was using gpt-4.1 but it almost always do that with gpt-4.1-mini.

Notice that first Responses API call of Sales Agent calls functions, but also generates a response without waiting the functions to be called. Then functions are called but...

AGENT FLOW

SALES AGENT -> Responses API response

SALES AGENT -> response of one of the tool calls

[seratch]

Thanks for writing in. Do you mean the last one in the trace is "get_related_knowledge_source" tool call and no responses api call to generate the final output after that? In this case, how does the final output look like? Is it just the output returned from the "get_related_knowledge_source" tool? This usually does not occur, so we need more information on what happened and how we can reproduce it.

Also, if this only happens with the mini model, your prompt may need to be more helpful for the mini model. Being more specific on what should be done would be helpful for cheaper models like mini and nano.

I do understand you're unable to share the actual production code, but if you could take time to share simpler code snippet to reproduce the situation, it would be greatly helpful for us to better understand.

[bakikucukcakiroglu]

In this example, I enabled parallel tool calls, which is why you see multiple calls. Here’s the sequence:

1. The Triage Agent makes an API call, and the API returns a handoff tool call (see “AGENT FLOW”).
2. The SDK executes that handoff tool call, transferring control to the Sales Agent (see “AGENT FLOW”).
3. The Sales Agent makes an API call, and the API responds with a structured output plus four tool calls (see “SALES AGENT → Responses API response”).
4. The SDK executes those four tool calls, and their outputs appear in the logs (see “SALES AGENT → response of one of the tool calls”).
5. The output from step 3 is then returned as the final output, yielding:

You’d expect one more API call that passes along the results of those four function calls—but it never happens.

To reproduce, try returning an assistant message and one or more tool calls together in the API response. My suspicion is that the SDK checks for a regular assistant message before honoring toolUseBehavior. Typically, a function-call response wouldn’t include a normal message, but some models prepend a reasoning-like message:

“Thanks for clarifying! You’re asking if the AUDIOCULAR – D07 CX31993 pairs well with the KZ Castor. Let me check the compatibility and details for you.”

I haven’t checked it fully, but the following snippet seems to contradict the intended run_llm_again logic.

// runImplementation.ts line 568
// we will use the last content output as the final output
const potentialFinalOutput =
  messageItems.length > 0
    ? getLastTextFromOutputMessage(
        messageItems[messageItems.length - 1].rawItem,
      )
    : undefined;

If models returning both a message and function calls is valid behavior, please clarify how the SDK will handle that. Otherwise, this needs to be fixed.

See related discussion: https://community.openai.com/t/responses-api-returns-message-function-call/1293055/15

I hope, this is helpful. If you need more confidential details (like traces, code) you can send me an email to baki@lookfor.ai

[bakikucukcakiroglu]

Any updates on this? @seratch

[seratch]

Still looking at other issues and pull requests, so I don't yet have time for checking this. Will update once I find something to share.

[seratch]

I've looked into this issue, but I haven't been able to reproduce the same situation yet. Adjusting runImplementation.ts is perhaps necessary, but I need more information to reproduce the issue.

[bakikucukcakiroglu]

Actually this issue is not that complicated from my point of view so I don't know what information I can provide. The SDK doesn't run the LLM again if LLM produces an output together with function calls in the first run(I don't know if LLM producing an output with function call is an error or intended behavior but it is not really important). So in the first occurrence of this problem in a conversation user mostly sees an answer that is created without function call outputs. And these responses are mostly full of (empty) promises like "I will create the invoice and get back to you when it is ready.". Actually the model also produces a function call for creating invoice with this output, and it is also executed but not used by the agent.

Then if the user asks "You said you will send me my invoice, I'm waiting for 1 hour". LLM tries to respond but now it fails because in the conversation history there is a function call but no corresponding function call output even though the functions executed in the previous run.

Did you try replicating it by a mock LLM response returning an output and a function call?

This is the exactly the same issue in openai-agents-python. Please especially see openai/openai-agents-python#1061 (comment)

This issue in the Community Forum also has a similar logic

This is too important. This is a problem of either the model (GPT4.1-mini) or the SDK but in any case you should take a quick action. See how we lose reputation.

Please communicate with the model team and first clarify if (output + function_call) behavior is intended or an error. Then act accordingly.

[bakikucukcakiroglu]

Hello @seratch ,

Could you try replicating it? I know there are too many issues but it's been 2 weeks and this issue causes vital problems. Once it gives a (tool call + output), whole conversation is dead.

[seratch]

We’re sorry about the disruption. I’ve reported this to the Responses API server team so they’re aware too. I'll take more time to explore what we can do on the client-side SDK, though ideally the issue will be resolved server-side as soon as possible. We appreciate your patience and will update you as soon as we have more to share.