google-genai: Streaming mode returns empty tokenUsage object

[Zamoca42]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Steps to Reproduce

1. Initialize a ChatGoogleGenerativeAI instance with streaming mode enabled (streaming: true).
2. Call a public method like invoke and pass callbacks with handleLLMEnd that includes a console.log to inspect the llmOutput field.
3. Observe that the estimatedTokenUsage object in the llmOutput field is empty ({}).

Example code to reproduce:

  const model = new ChatGoogleGenerativeAI({
    model: "gemini-1.5-flash",
    temperature: 1.0,
    streaming: true,
  });

  const result = await model.pipe(parser).invoke(formattedPrompt, {
    callbacks: [
      {
        handleLLMEnd: (output) => {
          console.log(output);
        },
      },
    ],
  });

Error Message and Stack Trace (if applicable)

{
generations: [ [ [ChatGenerationChunk] ] ],
llmOutput: { estimatedTokenUsage: {} }
}

Description

When using the ChatGoogleGenerativeAI class in streaming mode, the tokenUsage object is always returned as an empty object, even though the usage metadata is expected to be updated during the streaming process.

From analyzing the code, it appears that tokenUsage is not being updated within the _generate method in streaming mode. The chunk data from _streamResponseChunks includes usageMetadata, but it is not being added to the tokenUsage object.

System Info

node v20.18.1

[dosubot]

Hey there, @Zamoca42! 👋 I'm here to help you with any bugs, questions, or contributions you have in mind. Let's tackle this together!

There is a similar closed issue regarding the tokenUsage field being unavailable when streamRunnable is set to true in AgentExecutor. The suggested solution was to ensure that the model configuration includes the necessary settings to track token usage, specifically setting the streamUsage option to true. However, it was noted that even with this configuration, output.llmOutput?.tokenUsage remained undefined when streamRunnable was true, but token usage details appeared in output.generations ^[1].

Additionally, the pull request Feat/streaming token usage addressed token usage reporting in streaming mode by implementing callback handlers that capture token usage data. This ensures that token usage is accurately reported in both streaming and non-streaming modes ^[2].

In your case, it seems that the usageMetadata is not being correctly accumulated during the streaming process. The logic for accumulating the token usage across multiple chunks might not be correctly implemented, leading to an empty estimatedTokenUsage object ^[3]. You might want to verify if the streamUsage option is correctly set and if the callback handlers are properly capturing the token usage data.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[anadi45]

@Zamoca42 are you working on the fix? If not then I can

[Zamoca42]

@anadi45 I'm not working on it, please fix the issue!