Use tool output as agent output

[TimoVink]

Is there some equivalent to AutoGen's reflect_on_tool_use=False? That is: Some way for an agent to use the output of one of its tools as its own output, without needing another roundtrip to the LLM?

Worked example:

# Untested code, just illustrative

class LoremModel(BaseModel):
    paragraph: str

@function_tool
def generate_lorem(n: int) -> List[LoremModel]:
    '''Generate `n` paragraphs of lorem ipsum'''
    # omitted

agent = Agent(
    name='Lorem Agent',
    tools=[generate_lorem],
    output_type=List[LoremModel],
)

await Runner.run(agent, 'Generate 1000 paragraphs of lorem ipsum')

I believe the way this works is:

1. Prompt gets sent up to the LLM
2. LLM responds requesting invocation of generate_lorem(1000)
3. We execute that function and now have 1000 paragraphs of lorem ipsum in memory
4. We send these 1000 paragraphs back up to the LLM
5. The LLM generates a final response to the user prompt, which is just those same paragraphs of lorem ipsum again.

In practice we already have our answer after step 3, but then have this extra roundtrip in steps 4&5, which can get expensive in terms of both tokens and time if the output from our tool is large.

Is there functionality or design pattern to work around this? I think it would really help push the composability of agents even further than you already have with this SDK!

Real world usecase: I have an agent whose goal it is to retrieve a chunky piece of content. It has ~5 tools at its disposal. The "main" tool actually retrieves the data, the other tools help the agent determine the correct parameters with which to invoke the main tool. I would like to then use this "data retrieval agent" inside other agents which can then use this dataset to do interesting things.

I can work around this by making my "data retrieval agent" only return the correct parameters with which to call the main tool, and leave the actual retrieval to other agents, but then it feels like my agent isn't encapsulating a nice isolated chunk of work.

Love the library so far, thanks for your work!

[TimoVink]

Apologies if there's mistakes in my understanding of how AutoGen works, or how this library works. New to the space (as many of us are, I imagine!)

[rm-openai]

That's a cool feature. Currently don't have support for it, but we'll talk about it and see if we can add it!

[LeonG7]

Sometimes I query a lot of web data through an agent, but the agent summarizes the web information, resulting in missing information. I hope to receive the complete information obtained through web search. Perhaps using an intermediate variable can achieve this task, but this solution is cool

[rm-openai]

@TimoVink - implemented in #203. Chime in there if you have any feedback!

[TimoVink]

Wasn't expecting anything so quickly. Thanks!

Had a play, and this definitely solves my usecase. Left 2 small comments on the PR but even if it was merged exactly as-is I'd be a happy user.

[SeeknnDestroy]

Hey @TimoVink @rm-openai

Thanks so much for raising and implementing this feature — it’s super helpful! 🙌

I’d love to try this out in my own project. Could you share any details or code samples on how to use the new tool_use_behavior feature with an agent? A quick example or some guidance on how to set it up would be really appreciated!

Thanks again for the quick turnaround and the awesome work!