Multi-agent handoff via TwilioRealtimeTransportLayer problem with “nose”

[ihebakermi10]

A SupervisorAgent with handoffs (BookingAgent, TimeAgent, etc.) works over a plain WebSocket, but when using TwilioRealtimeTransportLayer, the session crashes during the first handoff—nose in voice

[ihebakermi10]

import Fastify from 'fastify';
import dotenv from 'dotenv';
import fastifyFormBody from '@fastify/formbody';
import fastifyWs from '@fastify/websocket';
import {
  RealtimeAgent,
  RealtimeSession
} from '@openai/agents/realtime';
import { TwilioRealtimeTransportLayer } from '@openai/agents-extensions';

dotenv.config();
const { OPENAI_API_KEY } = process.env;
if (!OPENAI_API_KEY) {
  console.error('Missing OpenAI API key.');
  process.exit(1);
}
const PORT = +(process.env.PORT || 5050);
const fastify = Fastify();
fastify.register(fastifyFormBody);
fastify.register(fastifyWs);

// --- Define Sub-Agents ---

const bookingAgent = new RealtimeAgent({
  name: 'Booking Agent',
  instructions: 'You help users book seats or appointments. Confirm slot availability and finalize bookings.'
});

const timeAgent = new RealtimeAgent({
  name: 'Time Agent',
  instructions: 'You check and confirm available time slots for bookings or appointments.'
});

const triageAgent = new RealtimeAgent({
  name: 'Triage Agent',
  instructions: 'You handle general queries and hand off booking or time-related questions.',
  handoffs: [bookingAgent, timeAgent]
});

// --- Define Supervisor Agent ---

const supervisorAgent = new RealtimeAgent({
  name: 'Supervisor Agent',
  instructions: [
    'You are the main orchestrator.',
    'Route booking-related intents to the Booking Agent.',
    'Route time-check intents to the Time Agent.',
    'Otherwise, defer to the Triage Agent.'
  ].join('\n'),
  handoffs: [bookingAgent, timeAgent, triageAgent]
});

// Root route
fastify.get('/', async (request, reply) => {
  reply.send({ message: 'Media Stream Server is running!' });
});

// Twilio incoming-call
fastify.all('/incoming-call', async (request, reply) => {
  const twiml = `
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Say>O.K. you can start talking!</Say>
  <Connect>
    <Stream url="wss://${request.headers.host}/media-stream" />
  </Connect>
</Response>`.trim();
  reply.type('text/xml').send(twiml);
});

// WebSocket for media stream
fastify.register(async (fastify) => {
  fastify.get('/media-stream', { websocket: true }, async (connection) => {
    const transport = new TwilioRealtimeTransportLayer({
      twilioWebSocket: connection
    });

    // Use SupervisorAgent here for multi-agent orchestration
    const session = new RealtimeSession(supervisorAgent, {
      transport
    });

    await session.connect({ apiKey: OPENAI_API_KEY });
    console.log('Connected multi-agent session');
  });
});

fastify.listen({ port: PORT }, (err) => {
  if (err) {
    console.error(err);
    process.exit(1);
  }
  console.log(`Server listening on port ${PORT}`);
});
process.on('SIGINT', () => {
  fastify.close();
  process.exit(0);
});

[seratch]

Thanks for taking the time to report this issue! @dkundel-openai should be able to check it once he is back from his days off. Otherwise, I can look into this issue later.

[grigoris-adamopoulos]

Hello there, I am experiencing the same problem in Realtime Agent handoff, using TwilioRealtimeTransportLayer.
It seems that during the agent handoff, session is updated but key config like input_audio_format and output_audio_format are changed from g711_ulaw (Twilio's audio supported format) to pcm16. That causes the session to crash.

[kkarkos]

I can confirm - handoff using TwilioRealtimeTransportLayer returns noise

[ihebakermi10]

@grigoris-adamopoulos any solution ?

[kkarkos]

Looks like @dkundel-openai was already working on something which might be related to the issue #126

[grigoris-adamopoulos]

@grigoris-adamopoulos any solution ?

None at the moment and seems that there is no workaround.