fix(voice): report e2e_latency for the first reply after a handoff

[rosetta-livekit-bot]

Ports livekit/agents#7167.

Keeps the latest unanswered user-turn metrics on the session so the first speech created by a handoff or inline AgentTask can report e2eLatency. The turn is settled when a speech reports it or when its reply chain ends without an answer.

Source diff coverage

• livekit-agents/livekit/agents/voice/agent_activity.py: adapted to agents/src/voice/agent_activity.ts. Preserves pending-turn ownership, onEnter claim/decline behavior, say and generateReply handling, tool-reply propagation, and reply-chain termination using TypeScript naming, milliseconds-to-seconds conversion, and AsyncLocalStorage.
• livekit-agents/livekit/agents/voice/agent_session.py: adapted to agents/src/voice/agent_session.ts as a session-scoped internal MetricsReport for the latest unanswered user turn.
• tests/fake_vad.py: not applicable. agents-js has no shared fake-VAD counterpart; its voice tests inject timestamped user turns directly rather than reopening scripted VAD streams, so the Python cross-stream fake-clock fix has no target infrastructure to modify.
• tests/test_e2e_latency_handoff.py: adapted in full to agents/src/voice/agent_activity_e2e_latency_handoff.test.ts, preserving all 15 source scenarios with Vitest, the target FakeLLM, and timestamped ChatMessage turns.

Infrastructure gap

The source framework exports e2e_latency through an OpenTelemetry metrics SDK and lk.agents.turn.e2e_latency histogram. agents-js currently has no meter provider or metrics exporter, so this PR cannot export that histogram without introducing the target's missing metrics subsystem. Message e2eLatency and the existing lk.e2e_latency span attribute are ported.

Testing

• pnpm exec vitest run agents (152 files, 2,505 passed, 5 skipped)
• pnpm --filter @livekit/agents build
• pnpm build
• pnpm --filter @livekit/agents lint (passes with existing warnings)
• pnpm exec prettier --check "agents/src/**/*.{ts,tsx,md,json}"
• pnpm --filter @livekit/agents typecheck
• cue-cli runtime handoff drive attempted, but the configured LiveKit endpoint rejected the supplied API key with 401 Unauthorized - invalid API key, so runtime framework-event validation could not complete.

---

Ported from livekit/agents#7167

Original PR description

Problem: e2e_latency is computed from the user turn's metrics held in a local of the reply task, so the first reply after an agent handoff never gets it, and a tool reply after an inline AgentTask reports the whole sub-conversation as one latency. The slowest turns of a call are the ones missing from the message, the lk.e2e_latency span attribute, and the OTel histogram.

Fix: The session keeps the latest user turn no speech has answered, set where the reply task commits the user message. The tool reply reads it, a say() or generate_reply() created inside on_enter claims it, and it ends when a speech reports it, when the reply chain ends without a tool reply or a handoff, or when an on_enter returns without speaking.

Closes #7157. Supersedes #7161.

Context for reviewing and coding agents

How to see it

tests/test_e2e_latency_handoff.py has 15 cases; most fail on main. Each asserts e2e_latency == started_speaking_at - user.stopped_speaking_at on the assistant message that answers a turn, and its absence on the message that does not. The inline-task case is the flow user -> tool -> await AgentTask -> the task asks -> the user answers -> the task completes -> tool reply: the question carries the first turn's latency and the tool reply carries the answer turn's.

Which speeches take the pending turn

The tool reply in _pipeline_reply_task_impl reads it in place of the threaded user_metrics if not forwarded_text else None. The two are equal on the paths main covers, since a step that spoke has already cleared it. say() and _generate_reply() claim it at creation, only under the existing _OnEnterContextVar gate, so a nested AgentTask awaited from on_enter claims it in its own on_enter, and a reply created after that await claims whatever turn the task left open. A generate_reply() outside on_enter, such as an inactivity nudge or a tool's background task, never reads it. The realtime path reports nothing, as on main.

When the turn ends unanswered

Three places: the report block of the speech that owns it, by identity, so an interrupted reply leaves it for the next speech; the end of _pipeline_reply_task_impl when no tool reply follows, the session's agent is unchanged, and the agent is not an AgentTask that just completed; and the finally of on_enter, for the turn that was pending when on_enter started, so a turn the user commits while on_enter runs stays pending for its own reply chain. The AgentTask.done() check exists because complete() only resolves a future and the switch back to the awaiting tool happens later, so the inner activity's chain end must not drop the turn the outer tool reply will answer.

Why a session field and not the task-local ContextVar of #7161

The value has to cross activities. An on_enter task has no speech handle, and an inline task's on_enter runs while the outer tool is still awaiting, so neither a handle attribute nor the outer task's context reaches it. #7161 also carried the value unconditionally, so an LLM that spoke before the handoff call produced two e2e_latency values for one user turn, and consumed it in any _generate_reply, so a say() greeting left it for a later unrelated reply.

Why a speech spawned from on_enter that speaks later reports nothing

Once on_enter returns without speaking, the agent has declined the turn. A speech from a task it spawned is a new utterance, and the on_enter end clear is what keeps a handoff nothing in the conversation caused from reporting a latency from minutes earlier.

Test harness change

FakeVAD restarted its clock per stream, and only the first stream ever saw an audio frame, so a spoken turn after a handoff had no timing anchors in any test. It now shares one clock across streams and waits for a frame only once. The 19 unit modules that use it pass unchanged.

[changeset-bot]

🦋 Changeset detected

Latest commit: 58a3ccb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 38 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Devin Review found 3 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

[devin-ai-integration]

🟡 Realtime handoff latency is lost

With a realtime model, takeOnEnterUserMetrics consumes the pending turn before selecting the reply path. The realtime path never receives it, so the first handoff reply omits end-to-end latency.

Prompt for agents

Preserve the metrics returned by takeOnEnterUserMetrics for RealtimeModel replies. Pass them through realtimeReplyTask and realtimeGenerationTask, then attach e2eLatency and ATTR_E2E_LATENCY when the first realtime output starts. Ensure a handoff onEnter reply consumes the pending turn only once, matching the pipeline and say paths.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Older tools erase newer latency

When a newer turn starts while an older tool runs, the older chain can later clear _unansweredUserMetrics unconditionally. The newer reply then loses its end-to-end latency.

Prompt for agents

Make chain termination clear only the metrics owned by that reply chain. Retain the original userMetrics identity throughout _pipelineReplyTaskImpl and compare the session field before clearing it, including the toolOutput.output.length === 0 branch. Add an overlap test where a second user turn commits while an older tool is still running.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Failed greetings consume pending latency

takeOnEnterUserMetrics removes the pending turn before the selected speech produces output. If that speech fails or gets interrupted, a later greeting cannot report latency for the unanswered turn.

Prompt for agents

Model pending metrics as claimed rather than permanently removed at speech creation. Restore the claim when the owning say or generateReply task ends without starting output, while avoiding restoration if a newer user turn has superseded it. Add tests for interrupted and failed first onEnter speeches followed by a successful greeting.

---

Was this helpful? React with 👍 or 👎 to provide feedback.