llm.RealtimeError("generate_reply timed out.") for "failed to generate a reply: generate_reply timed out" cannot be captured by exception

[moz164164]

Bug Description

According to (https://deepwiki.com/search/after-failed-to-generate-a-rep_f1da2841-db51-441a-ad41-86e5f2ebc777?mode=fast, when using the standard AgentSession.generate_reply() method, the framework handles the timeout internally and logs it instead of raising it to our application code.

However, when geneate_reply timeout happen, RealtimeError exception cannot be captured through:

try:
            await agent_session.generate_reply()
except RealtimeError as e:
            LOGGER.exception(
                "Error generating reply: %s"}
            )

Therefore, we cannot easily notice when the timeout happen and do some following action. I think this is the bug due to imperfect design of livekit agent.

Expected Behavior

Raise RealtimeError when timeout happen during await agent_session.generate_reply().

Reproduction Steps

1. Reduce the default timeout of the corresponding source code from 10 to 0.01 in `livekit/plugins/openai/realtime/realtime_model.py`
2. Use openai realtime model with openai.realtime.RealtimeModel(
                    model="gpt-realtime",
                    voice="marin",
                    turn_detection=ServerVad( 
                        type="server_vad"
                        prefix_padding_ms=300,
                        silence_duration_ms=500,
                        threshold=0.5,
                        create_response=False,
                        interrupt_response=False,
                    ),
                    temperature=0.6,
                    input_audio_noise_reduction=NOT_GIVEN
                    ),
                    input_audio_transcription=AudioTranscription(language=lang, model="whisper-1"),
                    max_session_duration=55 * 60,  
                )
3. Implement
 
try:
            await agent_session.generate_reply()
except RealtimeError as e:
            LOGGER.exception(
                "Error generating reply: %s"}
            )

Operating System

MacOS, linus

Models Used

gpt-realtime

Package Versions

"livekit~=1.1",                                                       
"livekit-agents[azure,openai,turn-detector,silero,elevenlabs]==1.6.0",
"livekit-api~=1.1",
"livekit-plugins-noise-cancellation~=0.2.0"

Session/Room/Call IDs

No response

Proposed Solution

Additional Context

No response

Screenshots and Recordings

No response