Text-only mode gives only empty responses

[andreysamode]

To reproduce:

git clone https://github.com/openai/openai-realtime-agents
cd openai-realtime-agents
rm -rf .next
rm -rf .node_modules
npm i
(add key to .env)
(in App.tsx, change "modalities: ["text", "audio"]," to "modalities: ["text"],"
next dev

Result:

The responses I get from the server are empty - just the timestamp:

Works fine with audio added to in modalities, both in text and audio.

Is there some extra bit of configuration that I need to do to make it work in a text-only mode?

[andreysamode]

To solve this, in src/app/hooks/useHandleServerEvent.ts, I changed:

    case "response.output_item.done": {
      const itemId = serverEvent.item?.id;
      if (itemId) {
        updateTranscriptItemStatus(itemId, "DONE");
      }
      break;
    }

to

    case "response.output_item.done": {
      const itemId = serverEvent.item?.id;
      if (itemId) {
        updateTranscriptItemStatus(itemId, "DONE");
        if (serverEvent.item?.content?.[0]?.type === "text") {
          const itemText = serverEvent.item?.content?.[0]?.text || "";
          updateTranscriptMessage(itemId, itemText, false);
        }
      }
      break;
    }

Not sure if it's the best way to do this, but it works.

[paix26875]

I found a way to resolve it. You can fix the problem by replacing:

case "response.audio_transcript.delta":

with:

case "response.text.delta":

in src/app/hooks/useHandleServerEvent.ts.
This should ensure that text-only mode works correctly. Hope this is helpful! 🚀

[AbrahamNobleOX]

@andreysamode, @paix26875

I provided a better and flexible approach in the PR below. Let me know your thoughts:

Branch: https://github.com/AbrahamNobleOX/openai-realtime-agents/tree/enabling-text-only

PR: #33

[paix26875]

@AbrahamNobleOX
Thank you for working on this! I really appreciate it. I'll take a look at your PR. 🚀

[altairmn]

@paix26875 You can also consider using https://github.com/outspeed-ai/voice-devtools
It's the same as realtime-agents UI but supports more models

[nm-openai]

Yes that's right, if you want to operate in text-only mode the text comes in as response.text not response.audio_transcript as reported above. Since this repo is focused on demonstrating voice in/voice out, and I'm actively trying to limit complexity beyond that, I'd prefer to leave it as-is. Out of curiosity why are you trying to use the realtime model for a text use case? This can be done at a much lower cost (although with higher latency) with our standard text models like GPT-4.1-mini