feat(soniox): map TTS character timestamps to aligned transcripts by acharan-tech-200037 · Pull Request #7159 · livekit/agents

acharan-tech-200037 · 2026-09-07T18:06:13Z

Summary

Closes #6727.

The Soniox TTS websocket returns character-level audio timings when the start config asks for
return_timestamps, but the plugin neither requested them nor read them. It declared
TTSCapabilities(streaming=True) with no aligned_transcript, so sessions running
use_tts_aligned_transcript=True fell back to the SDK's constant speaking-rate estimate — and an
interrupted utterance recorded imprecise spoken text, even though the provider exposes the same
character-level data livekit-plugins-elevenlabs already maps.

This requests return_timestamps, folds the per-frame character arrays into whole words, and pushes
them through the emitter's aligned-transcript path.

Word durations in one 6.3s reply measured live ranged from 45ms ('the') to 562ms
('interrupts.'), with a 549ms pause between sentences — spread a constant rate cannot represent.

Changes

return_timestamps: bool = True on TTS, wired to both the start config and
TTSCapabilities.aligned_transcript, so the plugin can never claim a capability it did not
request. return_timestamps=False opts out.
_recv_loop maps each frame's timestamps before pushing that frame's audio, since the
emitter attaches pending timed words to the next frame it emits.
_to_timed_words splits the character buffer with the SDK's split_words
(split_character=True, so CJK and Thai split correctly) and holds the trailing word back until
more characters arrive — a frame can end mid-word.
The emitted chunks tile the text. Each runs to the start of the next word, so separators ride
with the word before them. perform_text_forwarding builds the reply with out.text += delta, so
bare word slices would store "Hellothere,this" as what the agent said. Timings still describe
the word alone, not the separator.
Malformed frames (missing or mismatched arrays) are dropped rather than mis-aligned.

Where this plugin differs from ElevenLabs

ElevenLabs uses one websocket context per utterance. This plugin does not: a single
SynthesizeStream spans several Soniox stream_ids, rotating on LLM idle gaps and on stream age so
the server's per-stream timeout cannot kill synthesis mid-sentence. Three consequences:

Timings restart per stream. Soniox times characters from each stream's own first sample, so a
straight port would send TranscriptSynchronizer (dt = start_time - pushed_duration) backwards
partway through a reply. Each stream is offset by the audio already emitted when it opened.
The offset cannot come from pushed_duration() alone. That counts neither audio still queued
in the emitter nor the tail frame it holds back, so it lags. A _Timeline shared across the
segment tracks the furthest timestamp seen, and a rotated stream starts from whichever is
further along. Measured live, this moved a rotation from 2.090s to 2.134s — exactly the end of
the previous stream's last word.
The separator between streams is lost. The sentence tokenizer hands each stream its leading
whitespace (" Then, after a long pause...") and Soniox normalises that away before timing the
text, so two streams concatenated to "one sentence.Then". It is taken back from the text the
stream was actually given, leaving scripts that do not space their sentences alone.

Replay safety

A stream that fails transiently is replayed on a fresh stream_id so the reply is not cut short.
That is only safe while the stream has produced nothing, and pushed_duration() answers that
question late: audio handed to the emitter is invisible until it becomes frames, so a spent stream
could look replayable and have its audio — and its already-published words — spoken twice.

_StreamData.produced_output records the fact directly, set the moment audio or timed words are
handed over, and gates the replay. Words are published as soon as they are complete, so nothing is
withheld and nothing can be lost. A stream that will not be replayed flushes the word it was
holding back, since no more characters are coming for it — on failure, and in _run's cleanup,
which is the path an interruption takes.

Testing

tests/test_plugin_soniox_tts.py — 23 tests, 19 new:

the real _send_loop sends return_timestamps, and omits it when disabled
the real _recv_loop reassembles "Hi the" + "re" into ["Hi ", "there"], shifted onto the
segment timeline, and marks audio-only frames as produced output
the chunks rebuild the reply exactly — on one stream, across a rotation, and through
_recv_loop
timed words reach frame.userdata[USERDATA_TIMED_TRANSCRIPT] end-to-end through the real
SynthesizeStream
a reply split across two streams stays on one non-decreasing timeline
a rotated stream starts from the timeline rather than from a lagging pushed_duration()
a stream that published words, or merely pushed audio, is never replayed
an interruption through SynthesizeStream.aclose() keeps the last spoken word
mismatched timing arrays are ignored; _to_timed_words hold-back, flush and leading-separator
behaviour

Every fix was mutation-tested: reverting it fails a test, and only the tests that should fail.

ruff format, ruff check and mypy are clean on the touched files.

Verified against the live API

Checked end-to-end against a real Soniox key, including the rotation path:

stream 1 opened at offset 0.000s
stream 2 opened at offset 2.134s
    1.659 ->   2.134  'sentence.'     <- last word of stream 1
    2.416 ->   2.561  'Then, '        <- stream 2 continues the timeline
    ...
    6.607 ->   6.998  'third.'
rebuilt: 'First the agent says one sentence. Then, after a long pause, it says another one. And finally a third.'

The server restarts its own clock for stream 2, and 'Then, ' still lands after the previous
stream's last word rather than back near 0.28s — with the chunks rebuilding the spoken text exactly.

The Soniox TTS websocket returns character-level audio timings when the start config asks for return_timestamps, but the plugin neither requested them nor read them. Sessions running use_tts_aligned_transcript therefore fell back to the SDK's constant speaking-rate estimate, so an interrupted utterance recorded imprecise spoken text even though the provider exposes the same data the ElevenLabs plugin already maps. Request return_timestamps, buffer the per-frame character arrays into whole words, and push them through the emitter's aligned-transcript path. The trailing word is held back until more characters arrive or the stream ends, since a frame can split a word. Unlike the ElevenLabs plugin, one SynthesizeStream here spans several Soniox stream_ids: it rotates on LLM idle gaps and on stream age. Soniox times characters from each stream's own first sample, so every timestamp is shifted by the audio already emitted when that stream opened - the baseline _open_stream already tracks - keeping one non-decreasing timeline across a rotation.

…ripts Review found two defects in the aligned transcript. _to_timed_words emitted bare word slices, so the chunks no longer tiled the text. The SDK builds chat history with `out.text += delta` in perform_text_forwarding, so a reply was stored as "Hellothere,this". Each chunk now runs to the start of the next word, carrying the separator with the word before it, while its timings still describe the word alone. A rotation dropped the separator too: the sentence tokenizer hands each stream its leading whitespace (" Then, after a long pause...") and Soniox normalises that away before timing the text, so two streams concatenated to "one sentence.Then". The separator is restored from the text the stream was actually given, so scripts that do not space their sentences are left alone. pushed_duration() counts neither audio still queued in the emitter nor the tail frame it holds back, so it placed a rotated stream about 40ms early, ahead of words already emitted. Timestamp offsets now take whichever of it and the running timeline is further along. The retry watermark still tracks the emitter alone: it answers whether any audio has been emitted since the stream opened, which is a different question.

A stream that fails transiently before any audio is counted is replayed on a fresh stream_id, so the reply is not cut short. Timed words pushed during the failed attempt stayed queued on the emitter and were attached to the replacement's first frame, in front of the replay's own words: a reply came out as "Hello there, this Hello there, this is the first sentence." push_timed_transcript cannot be taken back. The emitter's watermark does not notice on its own - a lone 10ms chunk sits entirely in the tail frame it holds back, so pushed_duration() still reads the baseline that permits the replay. Words are now held until the emitter counts audio past that baseline, or until audio_end proves the stream will not be replayed. The replay stays available for the case it was added for, and a replaced stream has pushed nothing to repeat.

acharan-tech-200037 · 2026-09-07T18:55:26Z

The tests / unit-tests failure was test_false_interruption_resume.py::test_teardown_does_not_resume_a_deferred_pause — unrelated to this PR (2573 passed, 1 failed). That test asserts on a 50ms wall-clock margin (asyncio.sleep(FALSE_INTERRUPTION_TIMEOUT + 0.05)) under pytest.mark.unit rather than virtual_time, so it is sensitive to runner load. This branch only touches livekit-plugins-soniox, and the failing module imports no TTS plugin. Reopening to re-run CI.

Timed words are held until the emitter counts a stream's audio, so a stream that can still be replayed publishes nothing the replacement would repeat. But a stream can fail after the emitter accepts its audio, which is exactly when a replay is refused: the held words then went out with the discarded stream, dropping speech the user had already heard from the aligned transcript and from an interrupted turn's history. A cancelled stream lost them the same way. Replay and release turn on one question, so _settle_stream now answers it once and acts on both. Reading the watermark a second time in _retry_stream could let the two disagree, which is how the words went missing; that check moves out, and register_stream hands back the stream data so the words outlive the stream they belong to.

Three review findings in a row came from the same proxy. Whether a failed stream may be replayed was answered with the emitter's pushed_duration(), which reports audio only once it has become frames. Audio handed over but not yet counted made a spent stream look replayable, so its audio and its already-published words could be spoken twice; the machinery added to withhold words until the watermark moved then created the opposite fault, words withheld for audio the user does hear, and lost when the stream was discarded - on failure, and on the cancellation that ends an interrupted turn. The plugin knows the answer exactly, so it now records it: produced_output is set the moment audio or timed words are handed to the emitter, and that alone decides replay. Words go out as soon as they are complete, and a stream that will not be replayed flushes the word it was holding back, since no more characters are coming for it. This removes the withholding buffer and the second watermark it needed.

_to_timed_words holds the trailing word back in case later characters complete it, and the stream flushes it once settled. An interruption never gets there: aclose cancels _run, whose cleanup unregistered the active stream and ended the segment without flushing. The last word the user heard was dropped from the transcript of the very turn that most needs it - "Hello there, this is the first" for audio that said "...the first sentence." _run now flushes the active stream before unregistering it. The buffer is empty unless the stream produced output, and a settled stream has already been cleared, so nothing can be published twice; the flush precedes end_segment(), which closes the frame these words ride on. The test drives it through SynthesizeStream.aclose, the path a real interruption takes, rather than calling _settle_stream directly.

Characters that finish no word publish nothing and may arrive without audio, so neither witness of produced output fired - yet they had already advanced the timeline the whole segment shares. The stream still counted as replayable, and the replacement would have started from the advanced value, past audio the failed attempt never produced. Moving the timeline is as unrepeatable as publishing a word, so it now marks the stream the same way.

Spending a stream because it had moved the shared timeline was too blunt. A frame can carry characters that finish no word: nothing reaches the emitter, nothing is irreversible, and a transient failure after it was still safe to replay - but the reply was aborted instead. Buffered characters now move nothing. The timeline advances where words are published, so a stream that goes on to be replayed leaves nothing for its replacement to inherit, and there is no saved value to restore. A stream that ends cleanly flushes the word it was holding, which is when those characters count. produced_output goes back to meaning exactly what it says: something the emitter cannot be asked to take back.

Timestamps whose arrays disagree were skipped silently, but the buffer still held the word that frame was going to finish. The next frame landed straight on it and joined the two sides of the gap: "bro" + "ps" was published as "brops", a word nobody spoke, carrying plausible timings. The skip now publishes what is already whole, drops the fragment beside the gap so the gap stays a boundary, and logs. Text from the bad frame is still lost - it cannot be trusted - but nothing is invented.

Dropping the buffered prefix stopped the two sides of a gap being joined into a word nobody spoke, but the far side was still published on its own: "ps", the tail of "jumps", went into the transcript as a word. A word tail is no more real than the splice was. Characters are now dropped until a boundary arrives, across as many frames as that takes, so alignment resumes on a whole word. The word the gap swallowed is lost, which is the honest outcome; nothing is invented. The test that missed this asserted each published word was a substring of what was spoken - and "ps" is inside "jumps". It now checks membership in the set of spoken words.

Waiting for whitespace after a skipped frame never resolves in a script written without it. A single malformed frame in a CJK or Thai reply discarded every remaining character, so the rest of the turn had no aligned transcript at all - worse than the fragment the resync was added to prevent. Resync now asks split_words whether a character can continue the word before it, the same judgement _to_timed_words makes. In those scripts each character is already a word, so alignment resumes on the next character and only the skipped frame is lost. A separator is consumed where there is one; a character that stands alone is kept. The word held in the buffer is published rather than dropped when nothing could have continued it. Where a gap ends mid-word in a space-delimited script the tail still goes: nothing in the stream distinguishes the end of the broken word from the start of a new one. A gap that ends on a boundary now costs nothing extra.

…ng it A malformed frame has unusable timings, but its characters were still spoken. Dropping them cut those words out of the turn's history, and the resync then dropped the broken word either side of the gap as well. Only the timings are missing, so only the timings are given up: the gap's text, together with the word it broke, is published as a TimedString with no start or end. The synchronizer estimates across untimed text, which is the behaviour aligned transcripts replace - now confined to the stretch that lost its timings instead of applying to the whole turn. The reply reconstructs in full again, in space-delimited and spaceless scripts alike, and a stream that ends inside a gap still hands over what it was holding.

_publish_untimed pushed the gap's text without the leading separator the stream was handed, and left emitted_any unset. The next timed word then took the separator instead, so the reply lost its whitespace at the gap and grew a second one further along: " Then, after all" came out as "Then, after all". Also removes a comment left behind by the previous commit. It still described dropping the buffered prefix and losing the interrupted word, which is the opposite of what the code now does - and is the behaviour that kept being reported against this function.

A region recovered from a malformed frame was published with no timings at all. The synchronizer builds its rate curve from timed points and reveals text only as far as that curve reaches, so text outside it is never forwarded: a reply ending in such a region had its last words held back from the caption for good. The region's bounds are known even though its characters' are not. It opens where the last published word ended, which also gives the synchronizer the anchor it needs to spread the preceding text. A region followed by more timed words is closed by the next one; a final region has nothing after it, so it closes on the audio the emitter has by then.

Closing the final untimed region used the emitter's duration, which lags the audio it has been handed. The region was spread over an interval shorter than the speech it describes, so captions ran ahead of the voice - the same lag this PR already works around for rotation offsets and replay, used here as if it were exact. The previous commit closed that region to keep captions from stopping short, but they never did. The synchronizer uses annotations only while playback is inside them, estimates once past the last one, and releases whatever is left when playback completes, so an open tail still arrives. Only the start is claimed now, which is a real word end and anchors the text before it. Nothing is claimed about the end, because nothing here knows it.

acharan-tech-200037 · 2026-09-08T15:47:54Z

@longcw @chenghao-mou can you please review and tell me if any changes required

acharan-tech-200037 requested a review from a team as a code owner September 7, 2026 18:06