Managing context during tool runs

[pjsgsy]

Hi - I have a big issue trying to manage context during long runs where the gframework internally manages all tool iterations.

Hoping someone can point me in the right direction

1. Is there an intended hook point between tool iterations? e.g., an IToolCallInterceptor, a middleware delegate, or
   an event on ChatClientAgent?
2. Can ChatClientAgentOptions expose a per-iteration callback (before/after each tool-call round)?
3. Is IChatReducer intended to fire more than once per RunAsync? The current behavior (once only, before the first LLM
   call) makes it useless for long tool runs.
4. Is there a DelegatingChatClient hook point that fires between iterations inside the native loop?

I added post-stream compression, but it's not enough, and I cannot see a way to implement a hook that will help prevent a single 15-tool run from killing the conversation.

Thanks.

[Ramón Solís Núñez (ramonsolis11)]

The key point is that the native tool iteration loop is not owned by ChatClientAgent itself. It is primarily handled by FunctionInvokingChatClient.

FunctionInvokingChatClient repeatedly calls its inner IChatClient, executes any returned AIFunction calls, appends the resulting tool messages, and then calls the inner client again until there are no more actionable function calls or another termination condition is reached.

Because of that, I would not use IChatReducer as a per-tool-iteration interception mechanism.

IChatReducer is better understood as history/context preprocessing before an agent/model invocation, rather than as middleware around every internal function-calling iteration.

If you need visibility or context management between model calls inside a long tool run, the most useful interception point is the IChatClient pipeline.

A simplified pipeline looks conceptually like this:

ChatClientAgent
      |
      v
FunctionInvokingChatClient
      |
      | internal loop
      |
      +--> Model call
      |
      +--> Tool execution
      |
      +--> Model call
      |
      +--> Tool execution
      |
      `--> Final model call
              |
              v
        Inner IChatClient

If you place a custom DelegatingChatClient inside the function-invocation layer, it can observe each model call generated by that loop.

For example:

public sealed class ContextManagingChatClient : DelegatingChatClient
{
    public ContextManagingChatClient(IChatClient innerClient)
        : base(innerClient)
    {
    }

    public override async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        ChatOptions? options = null,
        CancellationToken cancellationToken = default)
    {
        var managedMessages = CompressIfNeeded(messages);

        return await base.GetResponseAsync(
            managedMessages,
            options,
            cancellationToken);
    }

    private static IEnumerable<ChatMessage> CompressIfNeeded(
        IEnumerable<ChatMessage> messages)
    {
        // token counting / summarization / pruning strategy
        return messages;
    }
}

Then compose the pipeline so that the context-management client wraps the actual provider client before the function-invocation layer performs its repeated calls.

Conceptually:

FunctionInvokingChatClient
        |
        v
ContextManagingChatClient
        |
        v
Provider IChatClient

That ordering matters.

If your middleware wraps the FunctionInvokingChatClient instead:

ContextManagingChatClient
        |
        v
FunctionInvokingChatClient
        |
        v
Provider

you normally see the outer request, but not every internal model round performed by the function-calling loop.

So regarding your questions:

1. Is there an intended hook between tool iterations?

   I am not aware of a dedicated IToolCallInterceptor or per-iteration callback on ChatClientAgent.

   The practical extensibility point is the IChatClient pipeline.

2. Can ChatClientAgentOptions expose before/after iteration callbacks?

   Not currently, as far as I can see. That would be a reasonable feature request if the framework wants first-class lifecycle hooks around each native function-call round.

3. Should IChatReducer fire once per tool iteration?

   I would not rely on it for that. Reduction and function-loop interception are different concerns.

4. Can DelegatingChatClient intercept internal iterations?

   Yes, provided it is positioned on the inner side of FunctionInvokingChatClient, because the function-invoking client calls its inner client once per model round.

For long-running agents I would therefore treat context management as middleware:

Agent
  |
Function Invocation
  |
Context Budget / Compression Middleware
  |
Telemetry / Logging
  |
Provider Client

This also keeps token-budget enforcement independent from the agent implementation itself.

One additional safeguard worth using is FunctionInvokingChatClient.MaximumIterationsPerRequest. Even with compression, a hard upper bound prevents pathological tool loops from consuming an unbounded amount of context or cost.

So my recommended approach would be:

• use DelegatingChatClient for per-model-round context management;
• position it underneath the function-invocation layer;
• use IChatReducer for conversation/history reduction;
• enforce a maximum tool iteration count separately.

If the framework eventually adds a native before/after-tool-round callback, that would be cleaner for observability, but the IChatClient pipeline already provides a workable interception point today.