feat(vad): Add Silero VAD voice activity detection module#1
Open
dittops wants to merge 1 commit into
Open
Conversation
Implement acoustic-level voice activity detection using Silero VAD, complementing the existing text-based turn detection with real-time speech boundary detection. Key features: - ONNX Runtime integration for Silero VAD model inference - Context padding (64 samples at 16kHz) for temporal awareness - Configurable thresholds, durations, and sample rates (8kHz/16kHz) - Speech state machine with configurable hysteresis - Automatic model downloading from Silero repository - Feature-gated implementation with no-op stub when disabled Performance: - 0.22ms average inference time per frame - 148x faster than real-time - P99 latency: 333μs Test coverage: - Unit tests for configuration and state management - Integration tests with real downloaded audio - Performance benchmarks Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Summary of ChangesHello @dittops, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the gateway's audio processing capabilities by integrating a Voice Activity Detection (VAD) module. This new feature leverages the Silero VAD model via ONNX Runtime to intelligently identify speech segments within an audio stream. The primary goal is to optimize resource usage and improve responsiveness by ensuring that downstream processes, such as Speech-to-Text (STT), only receive relevant audio. The implementation is highly configurable, allowing fine-tuning of detection thresholds and durations, and includes a fallback stub for environments where the VAD feature is not enabled. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This is a comprehensive and well-structured pull request that adds a significant new feature: voice activity detection using Silero VAD. The code is well-organized into modules for configuration, asset management, and the core detector logic. The inclusion of extensive unit tests, integration tests with real audio, and performance benchmarks is excellent and greatly appreciated.
However, I've identified a critical issue in the model_manager.rs file regarding the interaction with the ONNX model. There appears to be a mismatch between the code's expectations for the model's input/output tensors (names, shapes, and sizes) and the actual structure of the official Silero VAD model specified in the configuration. This will likely lead to runtime errors. I've provided a detailed comment with suggestions on how to resolve this.
Additionally, I've left a few other comments with suggestions to improve maintainability and follow common Rust idioms, such as reducing code duplication and refining test configurations. Overall, this is a great addition, and once the critical issue is addressed, it will be a solid contribution.
Comment on lines
+20
to
+283
| #[derive(Clone)] | ||
| pub struct SileroState { | ||
| /// Combined LSTM state tensor [2, batch_size, 128] | ||
| pub state: ndarray::Array3<f32>, | ||
| /// Sample rate indicator (8000 or 16000) | ||
| pub sr: i64, | ||
| /// Context buffer for temporal padding | ||
| pub context: Vec<f32>, | ||
| } | ||
|
|
||
| impl SileroState { | ||
| /// Create new state for the given sample rate | ||
| pub fn new(sample_rate: u32) -> Self { | ||
| // Silero VAD uses state tensor with shape [2, batch_size=1, 128] | ||
| let state = ndarray::Array3::<f32>::zeros((2, 1, 128)); | ||
|
|
||
| // Context size depends on sample rate | ||
| let context_size = if sample_rate == 8000 { | ||
| CONTEXT_SIZE_8K | ||
| } else { | ||
| CONTEXT_SIZE_16K | ||
| }; | ||
| let context = vec![0.0f32; context_size]; | ||
|
|
||
| Self { | ||
| state, | ||
| sr: sample_rate as i64, | ||
| context, | ||
| } | ||
| } | ||
|
|
||
| /// Reset the LSTM state | ||
| pub fn reset(&mut self) { | ||
| self.state.fill(0.0); | ||
| self.context.fill(0.0); | ||
| } | ||
|
|
||
| /// Get context size for this sample rate | ||
| pub fn context_size(&self) -> usize { | ||
| if self.sr == 8000 { | ||
| CONTEXT_SIZE_8K | ||
| } else { | ||
| CONTEXT_SIZE_16K | ||
| } | ||
| } | ||
| } | ||
|
|
||
| /// ONNX model manager for Silero VAD | ||
| pub struct ModelManager { | ||
| session: Arc<Mutex<Session>>, | ||
| config: VADConfig, | ||
| /// Current LSTM state | ||
| state: SileroState, | ||
| /// Cached input names | ||
| input_names: Vec<String>, | ||
| /// Cached output names | ||
| output_names: Vec<String>, | ||
| } | ||
|
|
||
| impl ModelManager { | ||
| /// Create a new ModelManager with the given configuration | ||
| pub async fn new(config: VADConfig) -> Result<Self> { | ||
| let model_path = assets::model_path(&config)?; | ||
|
|
||
| info!("Loading Silero VAD model from: {:?}", model_path); | ||
|
|
||
| // Load model on blocking thread | ||
| let session = tokio::task::spawn_blocking({ | ||
| let model_path = model_path.clone(); | ||
| let config = config.clone(); | ||
| move || Self::create_session(&model_path, &config) | ||
| }) | ||
| .await | ||
| .context("Failed to spawn blocking task for VAD model loading")??; | ||
|
|
||
| // Cache input/output names | ||
| let input_names: Vec<String> = session | ||
| .inputs | ||
| .iter() | ||
| .map(|input| input.name.clone()) | ||
| .collect(); | ||
|
|
||
| let output_names: Vec<String> = session | ||
| .outputs | ||
| .iter() | ||
| .map(|output| output.name.clone()) | ||
| .collect(); | ||
|
|
||
| info!("VAD model input names: {:?}", input_names); | ||
| info!("VAD model output names: {:?}", output_names); | ||
|
|
||
| let state = SileroState::new(config.sample_rate); | ||
|
|
||
| Ok(Self { | ||
| session: Arc::new(Mutex::new(session)), | ||
| config, | ||
| state, | ||
| input_names, | ||
| output_names, | ||
| }) | ||
| } | ||
|
|
||
| fn create_session(model_path: &Path, config: &VADConfig) -> Result<Session> { | ||
| let mut builder = SessionBuilder::new()? | ||
| .with_optimization_level(config.graph_optimization_level.to_ort_level())?; | ||
|
|
||
| if let Some(num_threads) = config.num_threads { | ||
| builder = builder | ||
| .with_intra_threads(num_threads)? | ||
| .with_inter_threads(1)?; | ||
| } | ||
|
|
||
| let session = builder.commit_from_file(model_path)?; | ||
|
|
||
| Self::validate_model(&session)?; | ||
|
|
||
| Ok(session) | ||
| } | ||
|
|
||
| fn validate_model(session: &Session) -> Result<()> { | ||
| let inputs = &session.inputs; | ||
| let outputs = &session.outputs; | ||
|
|
||
| debug!("VAD model inputs: {}", inputs.len()); | ||
| for (i, input) in inputs.iter().enumerate() { | ||
| debug!(" Input {}: {} ({:?})", i, input.name, input.input_type); | ||
| } | ||
|
|
||
| debug!("VAD model outputs: {}", outputs.len()); | ||
| for (i, output) in outputs.iter().enumerate() { | ||
| debug!(" Output {}: {} ({:?})", i, output.name, output.output_type); | ||
| } | ||
|
|
||
| // Silero VAD should have: | ||
| // Inputs: input (audio), sr (sample rate), h (hidden state), c (cell state) | ||
| // Outputs: output (probability), hn (new hidden), cn (new cell) | ||
| if inputs.len() < 4 { | ||
| warn!( | ||
| "VAD model has {} inputs, expected at least 4. Model format may differ.", | ||
| inputs.len() | ||
| ); | ||
| } | ||
|
|
||
| if outputs.len() < 3 { | ||
| warn!( | ||
| "VAD model has {} outputs, expected at least 3. Model format may differ.", | ||
| outputs.len() | ||
| ); | ||
| } | ||
|
|
||
| Ok(()) | ||
| } | ||
|
|
||
| /// Run inference on an audio frame | ||
| /// | ||
| /// # Arguments | ||
| /// * `audio` - Audio samples (f32, normalized to [-1, 1]) | ||
| /// | ||
| /// # Returns | ||
| /// Speech probability (0.0 - 1.0) | ||
| pub fn predict(&mut self, audio: &[f32]) -> Result<f32> { | ||
| let frame_size = audio.len(); | ||
| let context_size = self.state.context_size(); | ||
|
|
||
| debug!( | ||
| "VAD inference: frame_size={}, context_size={}, sample_rate={}", | ||
| frame_size, context_size, self.state.sr | ||
| ); | ||
|
|
||
| // Build audio with context padding: [context | audio] | ||
| // Total size = context_size + frame_size (e.g., 64 + 512 = 576 for 16kHz) | ||
| let mut audio_with_context = Vec::with_capacity(context_size + frame_size); | ||
| audio_with_context.extend_from_slice(&self.state.context); | ||
| audio_with_context.extend_from_slice(audio); | ||
|
|
||
| let total_size = audio_with_context.len(); | ||
|
|
||
| // Update context buffer with last N samples from audio | ||
| let context_start = if audio.len() >= context_size { | ||
| audio.len() - context_size | ||
| } else { | ||
| 0 | ||
| }; | ||
| self.state.context.clear(); | ||
| self.state.context.extend_from_slice(&audio[context_start..]); | ||
| // Pad if audio was shorter than context size | ||
| while self.state.context.len() < context_size { | ||
| self.state.context.insert(0, 0.0); | ||
| } | ||
|
|
||
| // Prepare inputs - Silero VAD uses: input, state, sr | ||
| // Create values for each input as needed | ||
| let state_dim = self.state.state.dim(); | ||
|
|
||
| let inputs: Vec<(&str, Value)> = self.input_names.iter() | ||
| .filter_map(|name| { | ||
| let value: Option<Value> = match name.as_str() { | ||
| "input" => { | ||
| // Use audio with context padding | ||
| Value::from_array(([1usize, total_size], audio_with_context.clone())).ok().map(|v| v.into()) | ||
| } | ||
| "state" => { | ||
| let state_data: Vec<f32> = self.state.state.iter().copied().collect(); | ||
| Value::from_array(([state_dim.0, state_dim.1, state_dim.2], state_data)).ok().map(|v| v.into()) | ||
| } | ||
| "sr" => { | ||
| // sr should be a 1D array with single element | ||
| let sr_shape: [usize; 1] = [1]; | ||
| Value::from_array((sr_shape, vec![self.state.sr])).ok().map(|v| v.into()) | ||
| } | ||
| other => { | ||
| warn!("Unknown VAD input name: {}", other); | ||
| None | ||
| } | ||
| }; | ||
| value.map(|v| (name.as_str(), v)) | ||
| }) | ||
| .collect(); | ||
|
|
||
| // Run inference | ||
| let mut session = self.session.lock(); | ||
| let outputs = session.run(inputs)?; | ||
|
|
||
| // Extract probability - first output | ||
| let prob_name = if !self.output_names.is_empty() { | ||
| &self.output_names[0] | ||
| } else { | ||
| "output" | ||
| }; | ||
|
|
||
| let prob_tensor = outputs | ||
| .get(prob_name) | ||
| .context("No probability output from VAD model")? | ||
| .try_extract_tensor::<f32>() | ||
| .context("Failed to extract probability tensor")?; | ||
|
|
||
| let (_shape, prob_data) = prob_tensor; | ||
| let probability = prob_data.first().copied().unwrap_or(0.0); | ||
|
|
||
| // Update state from outputs - second output | ||
| let state_out_name = if self.output_names.len() > 1 { | ||
| &self.output_names[1] | ||
| } else { | ||
| "stateN" | ||
| }; | ||
|
|
||
| if let Some(state_value) = outputs.get(state_out_name) { | ||
| if let Ok(state_tensor) = state_value.try_extract_tensor::<f32>() { | ||
| let (shape, data) = state_tensor; | ||
| if shape.len() == 3 { | ||
| let d0 = shape[0] as usize; | ||
| let d1 = shape[1] as usize; | ||
| let d2 = shape[2] as usize; | ||
| if let Ok(new_state) = ndarray::Array3::from_shape_vec((d0, d1, d2), data.to_vec()) { | ||
| self.state.state = new_state; | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| debug!("VAD probability: {:.4}", probability); | ||
|
|
||
| Ok(probability) | ||
| } |
There was a problem hiding this comment.
There appears to be a critical mismatch between how the code interacts with the ONNX model and the actual inputs/outputs of the official Silero VAD model. This will likely cause runtime failures.
- State Tensors: The model from the provided URL expects two separate state tensors,
handc, both with a shape of[2, 1, 64]. TheSileroStatestruct, however, defines a single combinedstatetensor with a shape of[2, 1, 128]. Thepredictfunction then tries to use an input named"state", which doesn't exist in the model. - Audio Input Size: The model expects a fixed-size audio input tensor (e.g., 512 samples for 16kHz). This implementation constructs an
audio_with_contextbuffer ofcontext_size + frame_sizeand passes this variable-sized tensor to the model. This will cause an input shape mismatch error.
To fix this, the implementation should be updated to align with the model's actual signature. You might find the ort crate's official silero-vad example helpful as a reference.
I recommend the following changes:
- Modify
SileroStateto hold two separatehandctensors, each with the correct shape (e.g.,ndarray::Array3<f32>::zeros((2, 1, 64))). - Update
predict()to passhandcas separate inputs to the model. - Update
predict()to parse thehnandcnoutputs from the model to update the state. - Ensure the
inputaudio tensor passed to the model has the fixed size the model expects (e.g., 512), not a dynamically combined size.
Comment on lines
+6
to
+39
| /// Result of VAD processing for a single audio frame | ||
| #[derive(Debug, Clone, Default)] | ||
| pub struct VADResult { | ||
| /// Whether the current frame contains speech | ||
| pub is_speech: bool, | ||
| /// Speech probability (0.0 - 1.0) | ||
| pub probability: f32, | ||
| /// Whether speech just started (transition from silence to speech) | ||
| pub speech_start: bool, | ||
| /// Whether speech just ended (transition from speech to silence) | ||
| pub speech_end: bool, | ||
| /// Duration of current speech segment in milliseconds (if speaking) | ||
| pub speech_duration_ms: u32, | ||
| /// Duration of current silence segment in milliseconds (if silent) | ||
| pub silence_duration_ms: u32, | ||
| } | ||
|
|
||
| /// Trait for Voice Activity Detection implementations | ||
| pub trait VoiceActivityDetector: Send + Sync { | ||
| /// Process a single audio frame and return VAD result | ||
| fn process_frame(&mut self, audio: &[f32]) -> Result<VADResult>; | ||
|
|
||
| /// Reset internal state (call when starting a new audio stream) | ||
| fn reset(&mut self); | ||
|
|
||
| /// Get the current speech probability | ||
| fn speech_probability(&self) -> f32; | ||
|
|
||
| /// Check if currently in speech state | ||
| fn is_speaking(&self) -> bool; | ||
|
|
||
| /// Get the configuration | ||
| fn config(&self) -> &VADConfig; | ||
| } |
There was a problem hiding this comment.
The VADResult struct and the VoiceActivityDetector trait are duplicated here and in gateway/src/core/vad/detector.rs. This duplication can lead to maintenance issues, as any changes would need to be applied in both places.
To improve maintainability, these shared definitions should be centralized. I suggest moving them to a single location (e.g., detector.rs or a new types.rs file) and then publicly exporting them from the main vad/mod.rs file. The stub implementation can then import and use these shared types.
Comment on lines
+174
to
+176
| std::fs::create_dir_all(&vad_dir).unwrap(); | ||
| let model_file = vad_dir.join(MODEL_FILENAME); | ||
| std::fs::write(&model_file, b"fake model data").unwrap(); |
There was a problem hiding this comment.
The test function test_model_path_with_cache is an async function, but it uses blocking I/O operations from std::fs. This can block the thread of the Tokio runtime, which can lead to unexpected behavior or performance issues in tests.
It's better to use the asynchronous versions from tokio::fs in async contexts.
Suggested change
| std::fs::create_dir_all(&vad_dir).unwrap(); | |
| let model_file = vad_dir.join(MODEL_FILENAME); | |
| std::fs::write(&model_file, b"fake model data").unwrap(); | |
| tokio::fs::create_dir_all(&vad_dir).await.unwrap(); | |
| tokio::fs::write(&model_file, b"fake model data").await.unwrap(); |
Comment on lines
+19
to
+22
| #[cfg(feature = "vad")] | ||
| pub use vad::{SileroVAD, VADResult, VoiceActivityDetector, create_vad}; | ||
| #[cfg(not(feature = "vad"))] | ||
| pub use vad::{SileroVAD, VADResult, VoiceActivityDetector, create_vad}; |
There was a problem hiding this comment.
The cfg attributes for exporting VAD components are duplicated for both when the vad feature is enabled and when it's not. Since the vad module itself handles the feature-gating internally (by using a stub module), this duplication is unnecessary.
You can simplify this to a single pub use statement, which makes the code cleaner and easier to maintain.
Suggested change
| #[cfg(feature = "vad")] | |
| pub use vad::{SileroVAD, VADResult, VoiceActivityDetector, create_vad}; | |
| #[cfg(not(feature = "vad"))] | |
| pub use vad::{SileroVAD, VADResult, VoiceActivityDetector, create_vad}; | |
| pub use vad::{create_vad, SileroVAD, VADBackend, VADConfig, VADResult, VoiceActivityDetector}; |
|
|
||
| // Skip actual model tests if model is not available | ||
| // These tests require the model to be downloaded | ||
| #[cfg(feature = "vad_integration_tests")] |
There was a problem hiding this comment.
These integration tests are gated by a custom feature flag vad_integration_tests. This is non-standard and makes the tests less discoverable, as they won't be run with typical test commands.
The idiomatic way to handle slow or network-dependent tests in Rust is to use the #[ignore] attribute. This allows the tests to be part of the standard test suite but skipped by default. They can then be run explicitly with cargo test -- --ignored.
I recommend removing the custom feature flag and using #[ignore] on these tests instead.
Suggested change
| #[cfg(feature = "vad_integration_tests")] | |
| #[cfg(feature = "vad")] |
dittops
pushed a commit
that referenced
this pull request
Jun 3, 2026
… + router), not just linear The streaming executor handled only LINEAR chains — any branching fell back to batch. This generalizes it to the full fan-out tree so branching DAGs stream too: - `streamable_tree` replaces `linear_chain`: collects every node reachable from the start whose in-degree is ≤1 (a tree). A JOIN (any node with >1 incoming edge) is a true synchronization point — it cannot emit until all its inputs arrive — so such graphs still fall back to batch `execute_from`. That's a semantic boundary, not a missed feature. - `execute_streaming_from` now runs EVERY reachable node concurrently, each with its own input channel, plus a forwarder per node that routes the node's output stream to its downstream inputs: broadcast to all (split) or, when an edge carries a condition, only to matching downstream (router, evaluated per chunk via dag.evaluator). Terminal nodes deliver to the client sink as chunks arrive — so a `translate → [tts→audio, text_out→text]` split streams BOTH audio and text concurrently. Close cascade: feed root + drop executor-held senders ⇒ each non-root input is held only by its single parent forwarder ⇒ closes root→leaves. LIVE-VALIDATED: `dag_streaming_fanout_audio_and_text_live` — translate fans out to a TTS branch and a text-output branch; the client received **4 audio chunks + 4 text chunks** concurrently from the two terminals. (Assertion relaxed off Devanagari — Sarvam streaming sometimes romanizes; the fan-out delivery is the proof.) Closes follow-up #1: linear + split + router all stream; only JOIN batches (by design). lib 5844/0; clippy gate clean; chaos + dag_dataplane (batch path) unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 11, 2026
…B (was 1509ms HTTP) Generic core/tts/websocket.rs WebSocketTtsClient: persistent socket, detached recv task holding NO outer locks (channel-only), per-utterance CancellationToken map, flush=false text accumulation, TTFB stamped on first binary frame into the existing observer hooks. DeepgramAuraTTS over wss://api.deepgram.com/v1/speak (Speak/Flush/Clear ↔ speak/flush/clear), SSRF-validated endpoint overrides. Defaulted BaseTTS::speak_with_context (zero changes across 38 provider impls). Selected via the STANDARDIZED features.streaming flag in create_tts_standard; absent/unsupported falls back to HTTP with a warning. Mock e2e 5/5 (flow/ordering/TTFB/cancel-mid-stream/aggregation/SSRF). LIVE GATE through the full gateway: streaming loop PASS, profiler-measured tts_ttfb p50 514ms vs 1509ms batch — the #1 realtime lever landed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 12, 2026
…t + 10 majors - CRITICAL #1: fire_forced_speech_final no longer aborts the claimer's own JoinHandle when the hard-timeout task fires — the self-abort killed the user callback chain (egress→controller→run_turn→LLM) at its first pending await with the segment buffer already consumed (turn silently lost). - #2: TurnEvent::ResetAggregation is a documented no-op at the orchestrator — both speech_final paths already reset the segment in the processor before delivery; the unscoped VoiceManager::reset_turn_aggregation (racy against a freshly-armed segment) is removed. - #3: InterruptionState::extend_playout uses a fetch_update CAS — plain load→store could resurrect a pre-clear playout deadline racing reset(). - #4: forced-fire dedup compares NORMALIZED words (case-folded, punctuation stripped) — providers re-format late finals (smart_format), byte containment missed them and ran a double turn; genuinely new words still pass. - #5: new TurnEvent::BargeInMopUp — emitted only when user speech continues while the bot is STILL audible after the turn started (in-flight speak resolving post-clear); orchestrator re-runs the idempotent barge-in. - #6: segment-transcript split — STTResult.segment_transcript carries the full accumulated segment for TURN POLICY (turn_transcript()); .transcript keeps the provider's raw fragment so client egress sees no duplication. - #7/#8: standardized AudioData::playback_ms — provider duration_ms first, PCM16 byte math at the CHUNK's own rate second (stale session atomic gone), flat 250ms OVER-estimate for compressed (mp3 byte math marked the bot silent mid-sentence). - #9: bot-busy probe now ORs ConversationOrchestrator::has_active_turn() — the LLM TTFT thinking window kept the MinWords barge-in gate down. - #10: clear_tts treats provider clear() failure as warn-and-continue — the early return skipped the audio-buffer clear and state reset on barge-in. - #11: effective_wait_ms clamps ONLY the TTFS extension — the configured floor is never silently shrunk to 2/3 of the hard timeout. - NEW (found by the unmasked #11): the monotonic ms clock is clamped to >=1 — a segment armed in the process's first millisecond stored the 0 sentinel into segment_start_ms, making the next is_final restart the hard-timeout backstop (forced fire deferred indefinitely). Pins: self-abort survival (pending-await callback), normalized dedup, floor-never-shrunk, BargeInMopUp emission, playback_ms preference order, 0-sentinel clock, segment/raw transcript split assertions. Floor: 6053/0 lib (all features) + clippy clean. Live: deepgram_live_e2e 8/8, dag_multivendor_live_e2e 5/5 (Deepgram+Sarvam). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 12, 2026
…-rate + 11 majors CRITICAL #1: LiveKitConfig used ONE sample_rate for both directions, so the egress-only client_playback_rate silently re-rated remote-participant (user mic) delivery — STT got time-stretched garbage and the 16k VAD/smart-turn feed broke. Split: sample_rate (egress publish track) vs ingress_sample_rate (STT-pipeline-derived, threaded from the session's STT config). CRITICAL #2/#3: default-features build break (feature-gated resampler behind unconditional egress code) was already fixed in 247e738 (rubato unconditional); cargo check --lib (default features) is now a standing gate. - #6/#12: G.711 mulaw/ulaw/alaw no longer go through PCM16 byte math — new sniff::is_linear_pcm16/is_g711 predicates; egress seam passes companded formats through untouched; playback_ms uses 1 byte/sample for G.711. - #7/#10: compressed playback estimate = bitrate math (128 kbps assumption) instead of flat 250ms/chunk — burst trains no longer book phantom playout seconds (MinWords gate stuck up), large chunks no longer under-count. - #8/#11: StreamResampler::flush() + EgressAudio::flush wired at TTS complete — the buffered tail (word-final consonants) of every utterance is delivered before TTSPlaybackComplete; 0-byte frames are never sent. - #9: resampler chunk is now ~20ms at the INPUT rate (was fixed 1024 frames = 128ms gather at 8 kHz telephony) — VAD/smart-turn decision latency for non-16k clients drops to one frame. - #4: LiveKit ingress frames now flow through ONE ordered bounded channel + single forwarder (was: a spawned task per frame — concurrent receive_audio calls could reorder frames, scrambling resampler state and the STT byte stream); overflow sheds the newest frame with a warning. - #5: barge-in clear EPOCH (bumped under the TTS write lock) + VoiceManager::speak_if_epoch — a sentence pump that lost the lock race to a clear can no longer enqueue stale audio after it; conversation pump, fallback speak, and eager-confirm speaks are all epoch-gated. - has_active_turn gaps: eager-confirmed replies now register a real turn (probe stays up while they speak); run_turn ends its turn AFTER the fallback speak, and no early-return can skip end_turn. - client_playback_rate validated (8k-192k; invalid = loud disable, never a 0 Hz LiveKit track); non_interruptible_until_ms accumulation is now CAS. - Wire: OutgoingMessage::STTResult gains optional segment_transcript (full accumulated segment on multi-final/forced speech_finals — previously not exposed on the wire at all); openapi.yaml re-blessed. - Honest pins: clamp_clock pure-fn 0-sentinel pin; BOTH forced-fire claimers (detection + hard-timeout) survive a pending await; A-G2 matrix gap cell (ttfs<floor && floor>2/3·hard); G.711/bitrate duration pins; flush pins; ingress/egress rate-split pin; rate-validation pins; resampler no longer panics on unbuildable rate pairs (passthrough). Floor: 6084/0 lib (all features) + default-features check + clippy clean + conversation_loop 6/6 + chaos 8/8. Live: egress 101 real Deepgram chunks 24k->48k (~2x, flush-tolerant); deepgram STT roundtrip green. (Earlier 0-chunk failures were THIS test panicking inside the provider callback on a transient — now surfaced.) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 13, 2026
…ime correctness Confirmed-finding fixes (12 reviewed; 5 critical): CRITICAL #3/#4 — A-G5 mute deadlocks: - reset_strategies() no longer resets MUTE strategies (session-lifecycle, not per-turn) — wiping them deadlocked MuteUntilFirstBotComplete and no-op'd FirstSpeechMute. - MuteUntilFirstBotComplete is now driven by a SHARED latch flipped on the orchestrator's first completed bot turn (not signal-sampling), so a user who listens silently through a greeting is unmuted — no forever-mute. Documented: the strategy needs a proactive bot utterance (greeting/disclaimer-on-join). CRITICAL #2 — B-G6 summarize lost-update RMW: - ConversationHistory carries a mutation `version`; new history_snapshot_versioned + replace_context_if_unchanged make summarization a CAS: a turn that appended during the (up-to-5s) summary inference bumps the version, so the stale rewrite is REFUSED — concurrent appends/tool-pairs are never destroyed. The orchestrator now runs summarization DETACHED so it never blocks the STT/turn chain. Pinned: concurrent-append-survives + CAS unit test. Realtime (B-G2) correctness — reachable via the realtime WS handler: - #1: clear the playback estimate on ResponseDone so a post-completion CancelResponse cannot truncate a fully-heard item (deleting server context). - #6: format-aware bytes/ms (PCM16=48, G.711=8) — hardcoded 48 over-truncated telephony 6x; audio sample_rate now follows the format too. - #7: emits_user_turn_frames is true unless turn_detection is the explicit manual None variant (OMITTED => OpenAI keeps its server-VAD default ON). - #11: run_barge_in_sequence is MODE-AWARE — the input-buffer clear+preroll run only in manual mode; in server-VAD mode they re-trigger VAD and destroy user speech, so only cancel+truncate run. - #10: DAG realtime nodes (both builders) source the api_key via resolve_node_credential instead of an empty string. - conversation_log replayed on reconnect is now bounded (100 items). D-G9 (review): waav_tts_chars_total is now incremented on the PRODUCTION speak path (speak_if_epoch + speak_with_interruption, not just the unused speak()), and counts CHARS not bytes (was a ~3x overcount for Devanagari). Deferred (need a live OpenAI Realtime endpoint to validate, no key available): #5/#9 the session-scoped DAG S2S map is NOT wired into production — the legacy request-scoped path stays the validated default; #8 watch-counter response.done attribution under server-VAD auto-responses; #12 reconnect replay of truncated-response transcripts. These are unit-tested where possible and documented as experimental. Floor: 6131/0 lib (all features) + default-features build + clippy clean + conversation_loop 14/14 + flow_manager 7/7 + realtime 115/115.
dittops
pushed a commit
that referenced
this pull request
Jun 13, 2026
PIPECAT_FIX_PLAN A-G6, step 2: the PlaybackPump that meters the PlaybackQueue to the transport, plus the queued+in-flight protection model the brutal review (wf_e53f4d85) required. PlaybackPump (core/audio/playback_pump.rs): - spawns a task that pops the queue, delivers each chunk to a sink (the egress path), and PACES by playback_ms minus a lead so the transport buffer stays ≤ lead (bounded barge-in residual). Aborts its task on drop. - PlaybackEnqueuer: a cheap cloneable handle for feeding the pump from the TTS callback hot path without owning the pump. PlaybackQueue protection model (review #2): a unit must stay protected for its whole life — QUEUED and IN FLIGHT (popped and playing). pop_front now MOVES an uninterruptible unit from `uninterruptible_pending` to `uninterruptible_in_flight` (increment-before-decrement so the active sum never dips to 0); the pump calls mark_played once playout completes. has_uninterruptible_active() = pending || in_flight is the barge-in protection signal — the natural, playout-anchored bound (NOT a synthesis-clock deadline, which review #1/#3 showed expires while a disclaimer is still queued behind interruptible audio). 12 audio unit tests (queue FIFO/selective-clear/in-flight tracking/saturating mark_played; pump in-order delivery, mid-stream selective clear, idle resume). clippy clean. VoiceManager wiring (the flag, on_tts_audio pump build, clear_tts gate on has_uninterruptible_active, stop() teardown) + VM tests land next. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 13, 2026
…iew-fixed PIPECAT_FIX_PLAN A-G6, step 3 (final): wire the PlaybackQueue/PlaybackPump into the VoiceManager behind a default-off flag, with all four brutal-review (wf_e53f4d85) findings fixed. Wiring: - VoiceManager gains playback_pump + uninterruptible_playback (AtomicBool, env WAAV_UNINTERRUPTIBLE_PLAYBACK, off by default; set_uninterruptible_playback toggles it before on_tts_audio). - on_tts_audio: when enabled, builds a pump whose sink IS the user callback and the egress wrapper ENQUEUES each chunk (tagged by interruption_state. allow_interruption) instead of delivering directly. Disabled ⇒ no pump, the wrapper does user_cb(audio).await exactly as before — the validated immediate-delivery hot path is byte-for-byte unchanged when off. - clear_tts: selective barge-in gated on q.has_uninterruptible_active() (queued OR in-flight) — drops queued interruptible audio, keeps uninterruptible, leaves provider+transport untouched so the disclaimer plays; else a full clear. Review fixes: - #1/#3 (HIGH): dropped the synthesis-clock `non_interruptible_until_ms` deadline from the gate — it expired while a disclaimer was still queued behind interruptible audio (synthesis bursts faster than the pump meters playout). The pump's mark_played (playout-anchored) is the natural protection bound. - #2 (HIGH): protection now spans in-flight units (pop moves queued→in_flight), so a barge-in during a popped/playing disclaimer no longer takes the transport-flush path. New test ag6_in_flight_uninterruptible_survives_barge_in. - #4 (LOW): stop() drops the pump FIRST, before the fallible disconnects, so a disconnect error can't leak the pump task. - The fixed gate removed the test's need for a usize::MAX deadline override. 3 VM tests (pump engaged only when enabled; queued + in-flight disclaimer survive barge-in). lib floor 6162/0, default-features check clean, clippy clean. Note: manager.rs/tests.rs were reverted to HEAD by a review-agent git op mid-session; this reconstructs the wiring with the review fixes folded in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 16, 2026
…nfirmed) A 4-critic adversarial review workflow (12 findings, 7 confirmed real, 5 refuted) found genuine bugs in the Phase-0 scaffold; all fixed + regression- tested: - #1 (high) connect() returned BEFORE the connection was established and reported NOT ready — the consumer (handlers/realtime) sends audio immediately, gated on is_ready(), so the first handshake window of EVERY session's audio was silently dropped. connect() now blocks on a oneshot the supervisor resolves on first connect (sync-readiness contract, matching openai/client.rs). - #2 (high) the ReconnectGovernor permit was held across the dial-failure backoff SLEEP (process-wide slots occupied by sleepers → storm-control collapse). The permit is now scoped to the dial only. - #3 (high) the INITIAL connect participated in the circuit breaker + governor (a new session could be deferred/throttled). Resilience now gates RECONNECT dials only, like the reference. - #4 (high) restored the wall-clock `first_delta` anchor so barge-in truncates at min(elapsed, received) — was over-truncating (received-vs-received). - #5 (med) S2sEvent::InterruptedByServer now actually cancels + truncates in the supervisor (it holds protocol+transport) — was a playback-clearing no-op. - #6 (med) push_wire fails fast with NotConnected instead of silently buffering frames the live socket will never carry. - #7 (low) stale truncate doc/comment removed. +3 regression tests (synchronous readiness, fail-fast-when-not-connected, server-interruption cancel+truncate). 11 scaffold tests, 6108 lib tests green, clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 16, 2026
…l clones on the scaffold (Phases 2-3) Three new realtime providers, each a thin RealtimeProtocol that EMBEDS the live-validated OpenAiProtocol and DELEGATES every wire method to it verbatim — the OpenAI GA wire stays byte-identical for all shared parts (proven by per- provider delegation smoke tests); only the connect target / auth (and Grok's bootstrap event) are overridden. The wire was never re-implemented. Each is a newtype over RealtimeSession<P>, so all three inherit the scaffold's reconnect + barge-in + resilience for free. - Azure OpenAI Realtime: embeds OpenAiProtocol; overrides connect_spec (api-key HEADER not Bearer; resource from `endpoint`, deployment from `model`, full-URL normalization) + provider_id. Targets the GA endpoint /openai/v1/realtime (NO api-version) — WaaV sends the GA wire, which MS docs say the dated ?api-version=…-preview endpoints reject (review-fix #2). - Grok / xAI: keeps the raw model string; overrides map_server_event for the `conversation.created` bootstrap (→ SessionReady; else delegates) + Bearer auth. No Sec-WebSocket-Protocol header (xAI uses the subprotocol field only for browser ephemeral tokens; can break a strict server upgrade — review-fix #3). - Inworld: embeds OpenAiProtocol (session.created bootstrap delegates unchanged); corrected to the DOCUMENTED session endpoint /api/v1/realtime/session ?key=<session>&protocol=realtime + server-side HTTP Basic auth (base64 of the api-key), NOT the OpenAI-style /v1/realtime?model= + Bearer (review-fix #1). Inworld needs a backend-minted session id; until the REST session-create handshake lands (a Phase-5 RestHandshake follow-up) the id is supplied via `endpoint` and connect_spec errors loudly without it rather than dialing a dead URL. Registration (all 6 surfaces): builtin metadata+factory+inventory::submit, dispatch BuiltinRealtimeProvider+PHF map+counts(2→5), get_supported_realtime_ providers [openai,hume,azure,grok,inworld], handler api-key arms (+ azure endpoint injection), config azure_openai_api_key/azure_openai_endpoint/ grok_api_key/inworld_api_key (env+yaml+merge+zeroize-on-drop) + RealtimeConfig .endpoint. ServerConfig has no Default, so the 4 fields were added to every exhaustive struct literal across the test suite. Built by a delegated implement→brutal-review workflow; the review's 3 confirmed doc-grounded findings (above) were all applied + regression-tested. Oracle: core::realtime 156 → azure 8/grok 7/inworld 7 green; full lib regression 6136 passed/0 failed (was 6114); openai integration 14/0 (count 2→5); clippy 0. LIVE-VALIDATION of Azure/Grok/Inworld is key-gated (no creds held) and deferred; the OpenAI-wire reuse is byte-identically validated via the embedded protocol. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 16, 2026
The 10th realtime provider, completing the scaffold's transport architecture with
the last net-new transport: Amazon Bedrock's InvokeModelWithBidirectionalStream
(HTTP/2 event stream, NOT WebSocket). DE-RISKED + structurally mirrored from the
verified in-tree precedent core/stt/aws_transcribe/client.rs (AwsTranscribeTransport
already drives the same smithy event-stream shape).
BedrockBidi transport (transport.rs + ConnectSpec::BedrockBidi{model_id,region},
session.rs UNCHANGED — transport seam only): client built from aws_config default
chain (SigV4 creds, NO api-key) like AwsTranscribeSTT; send(OutFrame::Text(json)) ⇒
wrap json bytes into a BidirectionalInputPayloadPart fed through an mpsc→async_stream
input half (mirroring Transcribe's audio_stream); recv() ⇒ drain the EventReceiver
output half, unwrap each payload's bytes ⇒ OutFrame::Text. Audio is base64-in-JSON
(the stream is event-framed, not raw binary). Dep: aws-sdk-bedrockruntime 1.62
(resolved 1.132 — same SDK train as the in-tree aws-sdk-transcribestreaming);
aws-config/credential-types/smithy were already deps.
NovaSonicProtocol: build_session_config emits the Nova Sonic bootstrap event
sequence (sessionStart→promptStart{audioOutputConfiguration lpcm/24000/voiceId,
toolConfiguration}→SYSTEM contentStart/textInput/contentEnd→USER-AUDIO contentStart),
all {"event":{…}} wrapped, sharing one session-stable promptName. encode_user_audio
⇒ audioInput{base64}. Load-bearing subtlety caught from the docs: transcript
role+generationStage live on the preceding contentStart (NOT textOutput) — tracked
via interior mutability; FINAL⇒final/SPECULATIVE⇒interim. audioOutput⇒Audio,
toolUse⇒FunctionCall, contentEnd(INTERRUPTED)⇒InterruptedByServer,
completionEnd⇒ResponseDone. Keyless: from_config ignores api_key (AWS creds via
aws-config); the matrix gained a KEYLESS_PROVIDERS skip-set (asserts nova_sonic
CONSTRUCTS with an empty key while all others still reject empty).
Registered all surfaces; count 9→10; get_supported ends [...,ultravox,nova_sonic].
Review (workflow wf_51a0f917) — confirmed transport_contained (session.rs unchanged)
+ dep_compiles + oracle_passed; comment-fix applied (resolve_region doc matched to
code). DEFERRED as documented live-hardening follow-ups (unkeyed → can't validate;
non-blocking on the documented happy path): (#1 MEDIUM) graceful teardown
(contentEnd→promptEnd→sessionEnd) is not driven on disconnect — needs a scaffold
close-path hook; the Bedrock stream-close still triggers server cleanup. (#2 LOW)
textOutput applies the stashed contentStart role/stage without contentId match —
defensive only; Nova Sonic's documented flow is strict adjacent 1:1.
NO AWS creds held → UNIT-VALIDATED (event shapes verified verbatim vs AWS Nova Sonic
input/output-events docs; the live SigV4 Bedrock round-trip is the only unexercised
path). Oracle: nova_sonic 22, scaffold 22, full lib regression 6270 passed/0 failed
(was 6243), matrix + openai integration counts→10, clippy 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 16, 2026
…uite compile break Adds tests/realtime_full_integration.rs (6 tests) proving the realtime fleet is integrated with the WHOLE WaaV stack + feature-compatible across ALL 11 providers (key-free — exercises construction + config-plumbing, which are local/pure): - every provider CONSTRUCTS through the REAL registry seam global_registry().create_realtime(name, cfg) — the identical PHF→factory→ panic-isolated path the /realtime handler AND the DAG node both use — with a RICH RealtimeConfig carrying the full feature surface (turn_detection server_vad, input_audio_noise_reduction, transcribe_input, a get_weather tool, reasoning_effort, temperature, max tokens, modalities, instructions, voice; endpoint where azure/ inworld need it). Providers ignore what they don't support (graceful). - config-plumbing proof now calls the REAL build_realtime_config (made pub + re-exported) — every feature field survives RealtimeSessionConfig→RealtimeConfig. - impact-analysis assertions: STT registry still 30, TTS intact, realtime EXACTLY 11 — the realtime work was additive, cascade/STT/TTS untouched. RCA findings (from the integration workflow, verified): the S2S providers OWN server-side VAD/turn/LLM/TTS; WaaV plumbs turn_detection/noise/reasoning config to them uniformly (WaaV's own Smart-Turn/Silero + noise_filter serve the CASCADE/LiveKit paths, NOT /realtime). input_audio_noise_reduction is provider-side (OpenAI near/far_field wire). reasoning_effort is honored by OpenAI's two-tier mapping, ignored gracefully by others. The fleet IS integrated into the DAG: dag/nodes/ provider.rs RealtimeProviderNode/SessionRealtime builds a RealtimeConfig and calls the SAME create_realtime seam, routing provider audio into the cascade sink as DagOutput::Audio (downstream can't tell S2S from TTS). FIX (review wf_0563f756 #1 MEDIUM — a real regression hidden by `cargo test --lib` not compiling tests/): the gemini/ultravox/speechmatics provider commits added ServerConfig api_key fields to src but missed ~13 test files' exhaustive `ServerConfig {}` literals → `cargo test --all-targets` failed to compile (E0063). Added gemini_api_key/ultravox_api_key/speechmatics_api_key: None to 29 literals across 15 test files. `cargo test --all-targets --no-run` now compiles clean. Oracle: --all-targets --no-run compiles 0 errors; realtime_full_integration 6/0; --lib regression 6295/0 unchanged; clippy no new warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 16, 2026
…edrock mock round-trip (key-free hardening) DAG INTEGRATION — realtime-as-a-DAG-node is now feature-complete. RealtimeProviderNode ::create_session previously built a MINIMAL RealtimeConfig (model/provider/api_key, ..Default for the rest), so a realtime provider used as a DAG node never received turn_detection/noise/transcribe/tools/reasoning/instructions/voice/etc. New pure build_node_realtime_config() deserializes the node's `config` JSON into a RealtimeConfig (full feature surface), then overrides the authoritative fields (provider, model, api_key via the existing credential resolver). Matches the HTTP /realtime path's build_realtime_config. 3 tests (full-surface plumbing, authoritative override, env-credential resolution). Caught + fixed a LATENT BUG: RealtimeConfig::api_key was the only field of ~20 lacking #[serde(default)], so serde_json::from_value on any config blob omitting api_key (the common case) failed the missing-field and silently fell back to default — dropping the ENTIRE feature surface. Added #[serde(default)]. Review (workflow wf_2b7f9856) fixes applied: - #1 HIGH (SSRF): build_node_realtime_config cleared realtime_endpoint_override but NOT `endpoint` (also a redirect vector — azure resource / inworld session-id / speechmatics URL). Now clears BOTH, so an untrusted DAG config blob can't redirect the upstream (a DAG node needing an endpoint must get it from trusted server config, like the HTTP path injects azure_openai_endpoint). + an SSRF test asserting endpoint cleared. - #3 LOW: a single malformed config field silently dropped the whole surface to defaults; now logs a warn! with node_id/provider/error so it's diagnosable. - #2 MEDIUM (noted, not over-claimed): the config-BUILDING is unit-proven; whether the session-scoped DAG path is wired into every production DAG-init flow is a separate pre-existing DAG-subsystem concern, not changed here. NOVA SONIC BEDROCK MOCK (Task 2, succeeded — harness in /tmp/waav_live/, not committed): the REAL BedrockBidiTransportFactory completes a full credential-free Nova Sonic round-trip against a local mock — aws-config honors AWS_ENDPOINT_URL (zero code change), plaintext HTTP/1.1 over http://, and a faithful re-implementation of the AWS smithy event-stream binary framing (verified vs aws-smithy-eventstream + aws-sdk-bedrockruntime source). Result: textOutput("hello from mock nova") + audioOutput back through the production transport. So all 11 providers now have a credential-free live round-trip exercising the actual gateway transport code (10 WS via mock_upstream + nova_sonic via the Bedrock mock). Oracle: dag 143, full lib regression 6318 passed/0 failed (was 6315, +3), --all-targets compiles, clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dittops
pushed a commit
that referenced
this pull request
Jun 17, 2026
…heduled playout, reconnect hardening First wave of the SOTA perf/resilience optimization (plan PERF_RESILIENCE_PLAN.md, from gateway-internals + Pipecat v1.4.0 brutal review + measured baseline + SOTA survey). The #1 gap — a dead-but-OPEN socket (the gateway never pings clients) — is FIXED and live-proven: a SIGSTOP-frozen gateway (TCP ESTABLISHED, no FIN/RST) is now detected in ~2.5s and reconnects, where before it hung forever. D1 LIVENESS / ZOMBIE WATCHDOG + protocol assert + verify-after-reconnect: - TS (NEW ws/liveness.ts): Node native WS ping(5s)/pong-deadline(3s) → terminate+reconnect; browser stale-inbound(12s) timer; window online/offline + visibilitychange → immediate probe; verify-after-reconnect (ping proves the NEW socket carries data, Pipecat pattern). - Python: websockets ping_interval=5/ping_timeout=3 (the 20/20 default did NOT trip) + _verify_connection + a stale-inbound watchdog (12s). - Widget: browser stale-inbound watchdog + network-change probe. - protocol_version now ASSERTED on ready (was captured-but-inert): major mismatch → typed error, minor drift → config_warning. Removed the dead JSON ping/pong handlers. D2 PLAYER (TS + widget): rewrote to 20ms scheduled-chunk playout (playhead=max(currentTime,nextStart), tracked scheduled sources) + true barge-in (stop ALL scheduled + clear + send {clear}); loss-tolerant late-drop. LEVERAGES the gateway audio knobs the SDK now sends: tts_config.audio_out_chunk_ms=20 (server-side barge-in truncation) + client_playback_rate=sink-rate (player does ZERO downlink resampling). D3 AUDIOWORKLET (TS + widget): off-main-thread capture (NEW capture.worklet.ts) — 20ms frames, Int16 in-worklet, transferable buffer pool (zero per-frame alloc); ScriptProcessor kept only as a legacy fallback. Wires the previously-unused workletNode. D4 RECONNECT HARDENING (TS + Python): client-side quick-failure breaker (survival<5s ×3 → typed FATAL, stops the storm — bounds the SDK's OWN reconnect, distinct from the gateway's upstream breaker); fast first hop (1000→200ms) + decorrelated jitter, 30s ceiling + reset-after-stable; widget post-ready reconnect un-gated; Python no longer clears in-flight audio on reconnect; bounded close-handshake. D5 CONTROL-QUEUE UNIFICATION (TS + Python): wired the previously-unused Python MessageQueue into _send_json (control ops buffer-and-flush on reconnect instead of raising); bounded drop-oldest uplink audio shedder (mirrors the gateway's egress discipline). Design principle (verified): the gateway's heavy resilience (breaker/governor/replay/ resumption) is UPSTREAM-ONLY, not on the client wire — the SDK does NOT mirror it; on a client↔gateway drop the existing reconnect→re-send-config→flush-queue is correct+sufficient. Validation: TS tsc 0 + 226 vitest (+32); Python 381 (+27); widget tsc 0 + build + 40 node-test (+18); zero regressions, drift guards green, key-scan clean. Live-proven zombie detect→reconnect ~2.5s (TS). Tier-2 (connect-time/pooling/opus/jitter-buffer) follows. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Features
Core Implementation
vadfeature flag with no-op stub when disabledConfiguration Options
threshold: Speech probability threshold (0.0-1.0, default 0.5)min_speech_duration_ms: Minimum speech segment (default 250ms)min_silence_duration_ms: Minimum silence for end-of-speech (default 300ms)sample_rate: 8kHz or 16kHz supportedframe_size: 256 (8kHz) or 512 (16kHz) samplesPerformance
Files Changed
src/core/vad/- New VAD module (6 files, ~1,400 lines)src/config/yaml.rs- VAD configuration supportsrc/core/mod.rs- Module registrationCargo.toml- Feature flag and dependenciestests/vad_tests.rs- Unit tests (9 tests)tests/vad_integration_test.rs- Integration tests with real audio (3 tests)benches/gateway_benchmarks.rs- Performance benchmarksTest plan
cargo test --features vad vad_tests)Dependencies
ort(ONNX Runtime) - Already used by turn-detect featurendarray- Already in dependencies🤖 Generated with Claude Code