FR: Real-time capabilities

[christianliebel]

With the implementation of multimodal capabilities (#71), it might also be worthwhile to consider integrating multimodal real-time capabilities, such as those offered by existing APIs like Gemini 2.0's Multimodal Live API or OpenAI's Realtime API.

[domenic]

It'd be especially interesting to hear what kind of JavaScript APIs might make sense for this sort of thing.

[christianliebel]

I guess this would expand the API shape quite a bit. This would require new methods and events. Here's a superficial example:

const session = await LanguageModel.create({ realtime: true });
const conversation = await session.getRealtimeConversation();

session.addEventListener("itemadded", (evt) => console.log('LOG:', evt.data));
await conversation.start(/* specify input/output device, conversation parameters etc. */); 
await session.prompt("How are you?");
// LOG: { role: "user", content: "How are you?" } 
// (Speech output) I'm fine, thank you. What do you want to talk about today?
// LOG: { role: "assistant", content: "I'm fine, thank you. What do you want to talk about today?" } 

await conversation.update({ interruptResponse: true });

// (Speech input) What is the answer to the university?
// LOG: { role: "user", content: "What is the answer to the university?" } 
// (Speech output) I do not know what the answer to… [interrupted]
// LOG: { role: "assistant", content: "I do not know what the answer to" } 
// (Speech input) Sorry, I meant universe.
// LOG: { role: "user", content: "Sorry, I meant universe." } 
// (Speech output) The answer to the universe is famously claimed to be 42.
// LOG: { role: "assistant", content: "The answer to the universe is famously claimed to be 42." } 

await conversation.stop();

This approach may require an (additional) multimodal model with built-in support for speech input and output. It could also interplay with the Media Capture and Streams, Speech Recognition, and Speech Synthesis APIs.