[BUG] SmolAgents CodeAgent Truncates Input Before Tool Execution

[grahama1970]

Summary

SmolAgents' CodeAgent incorrectly processes input before invoking tools, leading to truncation of long text even when a tool is explicitly designed to handle it. This defeats the purpose of tools and prevents reliable execution in workflows requiring large text processing.
Steps to Reproduce

Define a Tool (e.g., SummarizationTool) that accepts long text and chunks it internally.
Register the Tool with CodeAgent.
Send a large text input via agent.run(prompt).
Observe the Issue:
    The input never reaches the tool in full.
    Instead, CodeAgent processes it first, truncating it to its own model’s context length.
    The remaining truncated input is then passed to the tool, defeating the purpose of handling long text.

Expected Behavior

CodeAgent should immediately pass the input to SummarizationTool, without truncating it or attempting to reason about it first.

Observed Behavior

CodeAgent processes the input first, leading to truncation.
Even if the tool is the only valid choice for execution, the agent still tries to analyze the text before passing it.
The tool only receives partially processed, truncated input instead of the full text.

Proposed Fixes

Honor System Prompt Rules
    If the system prompt explicitly states:

        "For any summarization task, immediately use the summarization_tool without processing the input first."
        The agent must respect this directive.

Provide a Direct Tool Execution Mode
    If the user specifies a tool explicitly in the request:

    {
      "tool": "summarization_tool",
      "args": { "text": "Very long text..." }
    }

    The agent should not attempt intermediate processing.

Modify CodeAgent to Respect Token Limits at the Tool Level
    If an input exceeds the agent’s context limit, the agent should pass it to the tool immediately instead of trying to process it first.

Workaround (Not Ideal)

Users currently have to bypass SmolAgents completely and call the tool manually:

summary = SummarizationTool().forward(long_text)

This makes SmolAgents useless for tool orchestration in cases requiring large text handling.
Impact

Critical usability issue for any tool requiring large input (summarization, document parsing, data extraction, etc.).
Prevents SmolAgents from handling real-world workloads where LLM context limits are a known constraint.

Reproducible Code

from smolagents import CodeAgent, LiteLLMModel
from summarization_tool import SummarizationTool # Custom tool

agent = CodeAgent(
tools=[SummarizationTool()],
model=LiteLLMModel(model_id="openai/gpt-4o"),
system_prompt="Always use summarization_tool for large text. Do not process text yourself."
)

long_text = "..." # Large document exceeding LLM context limit

response = agent.run(f"Summarize this:\n{long_text}")
print(response) # 🔴 Receives truncated text, tool never sees full input

Priority: HIGH

This blocks any real-world usage of tools that are supposed to handle large input. If SmolAgents cannot be trusted to delegate correctly, it cannot be used in production.
Request

🚀 Please fix this by ensuring CodeAgent does not process input that is meant for a tool before calling the tool.

I've include a GIST of the code here: https://gist.github.com/grahama1970/12e4051997f8988bfdd56796436a6cae

Much appreciation in advance

[sdrakulich]

I'll add some thoughts, as I have truly struggled with this a lot with very little [consistent] success.

Using the agent to answer based on a long context is...not a good time. Especially if the agent decides what the context is, and it usually only prints the first 500 characters. If it doesn't print, or it's not a return value from a previous tool, then it just fails.

A special class of Tool that has output_type = "response" might be the lowest-barrier way to implement this? That way, whatever is passed in as Args: or Inputs: is characterized within the prompts as the context to provide an answer about.

What do you think is the most elegant solution for this? Querying the agent beyond the task defined in .run() seems really important, For example, the agent's expected final_answer() is an analysis of prior processing the agent produced when using another tool.

Ultimately, you can accommodate this with the agent defining the Arg that simply gets passed through the forward() call, but in the end it still depends on the agent having printed sufficient context at an earlier step, to answer correctly.

I'm asking for a more deliberate prompt injection of context that happens if the expected output_type = "response". Curious to hear what the reasoning was to not enable the agent to answer in such a way, as it is impossibly hard to get the CodeAgent to produce the final_answer() that is based off producing context then reasoning over it, without resorting to some inception-like methods to have tool calls produce the "response" you wanted it to produce. Even then, it doesn't do it, since the Inputs/Returns coupling is so prominent, and the description to pass when doing such an operation is just so confusing within the broader understanding the agent has from all the prompts and injections as they currently exist.

Really appreciate the framework, but hoping for such a fundamental, necessary feature to be integrated in. Thank you for considering.