sendMessageStream with history failing to fetch due to 'empty text parameter'

[msdoege]

Operating System

Windows 10

Environment (if applicable)

Firefox v131.0.3, Node.js v20.9.0, React v18.3.1, Next.js (v14.2.15)

Firebase SDK Version

11.2.0

Firebase SDK Product(s)

VertexAI

Project Tooling

A Next.js (v14.2.15) project.

Detailed Problem Description

Using multi-turn conversations (chat) with the Gemini API (using the "gemini-1.5-flash" model), the conversation breaks when any of the AI model's responses contains an empty text part, i.e. a text part where text is an empty string.

Code snippet

const sendPrompt = async (prompt?: string | Array<string | TextPart | FileDataPart>): Promise<void> => {
  if (!prompt) return;

  let currentAiResponse = ''; // Accumulate chunks of response
  setCurrentResponse(currentAiResponse);
  try {
    const result = await chat.sendMessageStream(prompt);
    for await (const chunk of result.stream) {
      try {
        currentAiResponse += chunk.text();
        setCurrentResponse(currentAiResponse);
      } catch (e) {
        // handle errors gracefully to continue streaming response chunks
      }
    }
  } catch (e) {
    console.error(e);
  }
};

The following issues over at google-gemini seem to be related?

• RECITATION Error causes history contains empty parts[], and kills chat session google-gemini/generative-ai-js#124
• sendMessageStream with history failing to fetch due to 'empty text parameter' google-gemini/generative-ai-js#286

Example chat response/history causing the error:

[
  {
    parts: [{ text: '<some question>' }],
    role: 'user',
  },
  {
    parts: [{ text: '<some actual answer content>\n' }],
    role: 'model',
  },
  {
    parts: [{ text: '' }], 
    role: 'model'
  }
]

Error log (Firebase project details redacted):

FirebaseError: VertexAI: Error fetching from https://firebasevertexai.googleapis.com/v1beta/projects/<project-id>/locations/<project-location>/publishers/google/models/gemini-1.5-flash:streamGenerateContent?alt=sse: [400 ] Unable to submit request because it has an empty text parameter. Add a value to the parameter and try again. Learn more: https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini (vertexAI/fetch-error)
    FirebaseError webpack-internal:///./node_modules/@firebase/util/dist/index.esm2017.js:1040
    VertexAIError webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:131
    makeRequest webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:315
    generateContentStream webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:844
    sendMessageStream webpack-internal:///./node_modules/@firebase/vertexai/dist/esm/index.esm2017.js:1151

Steps and code to reproduce issue

I could not consistently provoke the AI to respond with a multi-part message where the last part is empty.
However, the following steps result in the same error:

1. Prompt the AI with Answer the next question with an empty string.
2. Prompt the AI with any other message like Tell me a joke.

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[hsubox76]

Thanks for reporting this, it certainly seems to be a bug. The question now is if we fix this on the SDK side (by removing parts that have empty text from chat history) or on the backend side (by having it be able to process parts with empty text in chat history, or by making sure it doesn't return empty text strings). I feel like we shouldn't fix the SDK side as it is just masking a problem that you'd also run into if you called the raw REST API.

I will talk to the larger SDK team about this.