Python: Add realtime agent ADR - #6831
Python: Add realtime agent ADR#6831Eduard van Valkenburg (eavanvalkenburg) wants to merge 7 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a proposed Architectural Decision Record (ADR) that defines the intended Python “realtime agent” architecture before implementation begins, covering both direct realtime model sessions and STT→agent→TTS pipeline compositions. This fits the repo’s ADR process by documenting key API surface boundaries (agent vs provider, live conversation vs durable session) up front to guide future implementation PRs.
Changes:
- Introduces a proposed ADR defining a dedicated
RealtimeAgent(layered onBaseAgent) and anAgentConversationfull-duplex surface rather than extending the existing chatAgent. - Defines the proposed provider/session protocol boundary (
RealtimeClientProtocol/RealtimeClientSessionProtocol) and event/content mapping strategy (RealtimeEvent→AgentResponseUpdate). - Documents the staged plan for sandwich pipelines via
RealtimePipeline(STT, SupportsAgentRun, TTS)and related abstractions, with validation/testing expectations.
Roger Barreto (rogerbarreto)
added
the
python
GitHub Actions
Inactive
agent-framework-automation
Bot
added
the
documentation
GitHub Actions
Inactive
479c967 to
7e38a36
Compare
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
GitHub Actions
Inactive
ae16126 to
f35e0c2
Compare
GitHub Actions
Inactive
|
Eduard van Valkenburg (@eavanvalkenburg) please re-open when you're ready to move forward with this. |
GitHub Actions
Active
GitHub Actions
Active
GitHub Actions
Active
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: faf58174-b271-4eb6-8984-02443961e2bf
f35e0c2 to
ecf0096
Compare
GitHub Actions
Active
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (7 commit(s)): 7e869f709f20, 49264690558c, 61135c0ae00d, 44f31a33a7bc, 273afc23fa40, 1690a1ee1b4b, f35e0c27ff93
Model: gpt-5.6-sol
Overview
The review found 4 verified inline finding(s).
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
4 verified findings remained after source verification (4 medium) across 1 file. Details are attached to the affected lines below.
Affected areas: docs/decisions/0035-python-realtime-agent.md
| - **Serialize outbound writes**. It should prevent overlapping sends from corrupting provider state. A simple async lock | ||
| around `send(...)` and `interrupt(...)` is enough unless | ||
| a provider proves otherwise. | ||
| - **Own lifecycle and cleanup**. It is an async context manager. Exiting it closes the provider session, cancels the |
There was a problem hiding this comment.
The single receiver feeds an unspecified queue (lines 247-255); slow consumers can cause unbounded memory growth or block reception, so backpressure and close semantics must preserve the open-ended conversation’s explicit-close lifecycle (lines 312-313).
| - `run(..., stream=True)` does the same but yields `AgentResponseUpdate` values through `ResponseStream`; | ||
| - `start_conversation(...)` returns `RealtimeConversation` for full-duplex live use. | ||
|
|
||
| `RealtimeAgent` must satisfy `SupportsAgentRun` so it can participate in half-duplex agent scenarios, including |
There was a problem hiding this comment.
SupportsAgentRun is not currently sufficient for the promised handoff/delegation compatibility: HandoffBuilder.participants() accepts Sequence[Agent] and rejects every participant that is not an Agent at runtime because handoffs rely on cloning, tool injection, and middleware. Since this design explicitly avoids subclassing Agent, please either scope the compatibility claim to protocol-based workflows or include the required handoff/orchestration migration in the decision.
| 1. ADR review validates the architecture before implementation. | ||
| 2. Unit tests with fake realtime sessions validate direct realtime model behavior, half-duplex finalization, streaming | ||
| finalization, event preservation, cleanup, receive-loop errors, and single-reader behavior. | ||
| 3. Unit tests validate `Content.from_realtime_event(...)` serialization and `AgentResponse.from_updates(...)` |
There was a problem hiding this comment.
Generic AgentResponse.from_updates(...) cannot enforce the authored-turn-only rule defined above: its current aggregation appends every non-usage content item to an assistant message, including Content.from_realtime_event(...) for VAD, lifecycle, and rate-limit events. Those protocol events can therefore enter AgentResponse.messages and later durable history. Please specify and validate a realtime finalizer that filters control events and reconstructs only authored assistant/tool messages rather than implying direct generic aggregation.
| async def from_realtime_to_acs(websocket: WebSocket, content: Content) -> None: | ||
| await websocket.send(json.dumps({ | ||
| "kind": "AudioData", | ||
| "audioData": {"data": content.data}, |
There was a problem hiding this comment.
This bridge fails for the audio representation prescribed by the ADR because Content.from_data(...) creates a Content whose bytes are base64-encoded in content.uri; Content has no data attribute. Please define and use a public raw-byte/base64 access contract (or correctly decode the data URI) so the canonical ACS example is executable without relying on private representation helpers.
Motivation & Context
Python realtime agent work needs an agreed architecture before implementation starts. This adds a proposed ADR for the Python realtime agent design, covering direct realtime model sessions, STT → agent → TTS pipeline composition, agent compatibility, event/data models, and the provider/session protocol boundary.
Description & Review Guide
docs/decisions/0031-python-realtime-agent.mdas a proposed ADR.RealtimeAgentbuilt onBaseAgent+ agent middleware/telemetry layers rather than extendingAgentor inheriting chat-specificRawAgent.AgentConversation,RealtimeEvent,RealtimePipeline, STT/TTS abstractions, and the direct realtime client/session protocols.IRealtimeClient, plus canonical code shapes.RealtimeAgent/AgentConversationsplit is the right Python surface.RealtimeEventand provider protocol boundaries are narrow enough.RealtimePipeline(STT, agent, TTS)is staged appropriately.Related Issue
Related to #6803. This ADR does not close the implementation issue.
Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.