feat(collection): add optional speech for WebXR and desktop controls

[TomCC7]

Contribution path

Requested collection feedback feature; no separate tracking issue.

Stack: builds on #4088 and targets cc/feat/abc-policy. The parent owns the shared HTTPS/Hugging Face asset downloader; this PR adds its Kokoro consumer and recording feedback.

Problem and behavior

Operators can hear spoken confirmation when recording starts, saves, or discards an episode, either in a WebXR headset or through the computer running attached collection controls. Speech is off by default.

• Headset: enable nested WebXR configuration with --tts.enabled=true when starting the blueprint.
• Desktop, including teach collection: attach with dimos imitation collect --tts. This command-local option does not change the running blueprint's configuration.

Collection owns phrase wording, confirmed-transition selection, and preparation in dimos/imitation/collection/prompts.py. WebXR and desktop controls share that helper. Before accepting controls, each enabled consumer prepares all three WAVs and releases the inference engine. Recording callbacks perform no inference. Initialization, attachment snapshots, duplicate events, unchanged status polls, and idle discard commands remain silent.

WebXR pushes selected WAV audio over the existing WebSocket; the browser only plays/cancels audio. Desktop controls use nonblocking sounddevice playback on the default output device. New speech interrupts old speech. Detach/disconnect stops playback, and playback errors leave collection controls usable.

Dependencies and assets

The optional tts extra uses official kokoro==0.9.4, PyTorch CPU inference, and official hexgrad/Kokoro-82M v1.0 assets at a pinned revision. The default voice is af_sarah. Missing speech assets download through the parent's shared helper into the dimOS XDG cache. Misaki/spaCy automatically installs the missing English tokenizer with pip or uv on first enabled preparation; installer exits become ordinary startup errors.

Cached preparation works offline once the tokenizer is installed. Exact uv sync can remove the lazily installed tokenizer; uv run --no-sync preserves the prepared environment. Desktop playback reuses the already-locked sounddevice package; adding it to the tts extra introduces no new locked packages or version changes.

How to test

uv sync --extra manipulation --extra tts

For headset speech, retain the usual recording, task, hardware and camera options:

uv run --no-sync dimos run dual-openyam-quest-collection --tts.enabled=true <your existing options>
uv run --no-sync dimos imitation collect

--armteleopmodule.tts.enabled=true is the fully qualified WebXR option. TTS module configuration remains limited to enabled and voice.

For desktop speech, start any supported collection stack (including openyam-teach-collection) with its usual options, then attach:

uv run --no-sync dimos imitation collect --tts

Headset and desktop playback are independently optional.

Manual checklist

• Headset: open /teleop, select Connect, enter XR; connecting is silent.
• Quest B or terminal Space from idle: “Recording started.”
• B/Space while recording: saved count increases and “Episode saved.”
• Start again, then Quest Y or terminal D: discarded count increases and “Recording canceled.”
• Idle discard and unchanged status polls: no repeated speech.
• Desktop teach collection: Space, Space, Space, D produces start, save, start, cancel prompts.
• Toggle quickly: newer audio interrupts older audio; controls remain responsive.
• Leave XR or detach/disconnect during playback: audio stops. Reattachment is silent, including mid-recording.
• Restart with prepared assets/tokenizer and outbound internet blocked: preparation succeeds.
• Disable both options: collection works silently without optional inference or playback imports.
• Record headset/browser and desktop audio-device details. Physical device validation remains pending.

See collection audio usage for setup and full manual instructions.

Validation

• 50 focused Python tests passed across collection prompts, TUI, desktop playback, Kokoro and WebXR after simplifying the tests. Earlier CLI routing, shared asset and logger convention checks also passed.
• Five JavaScript playback tests passed.
• Real Kokoro + Chromium smoke: offline preparation succeeded in 5.46 seconds; pushed WAV played with no speech HTTP requests, and reconnect received only the status snapshot.
• Desktop CLI smoke used real Kokoro WAVs, the real episode state machine, and headless terminal controls: four correct prompts, silent repeated polls, and playback stopped on detach. The physical speaker output was mocked.
• Earlier fresh-virtualenv/forkserver smoke verified automatic tokenizer installation into the worker's own environment and offline reuse.
• Targeted mypy and commit hooks passed. The parent fixes the asset downloader's logger convention failure.
• No robot motion, physical headset, or physical speaker test was performed.

AI assistance

Implemented and validated with Codex, including documentation and this draft PR.

Checklist

• I have read and approved the CLA.

Agent: Codex

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
5730	1	5729	51

View the top 1 failed test(s) by shortest run time

dimos.web.relay_bridge.test_relay_bridge_e2e::test_external_relay_with_real_certificate

Stack Traces | 0.215s run time

tmp_path = PosixPath('.../pytest-0/popen-gw2/test_external_relay_with_real_0')

    def test_external_relay_with_real_certificate(tmp_path: Path) -> None:
        # A relay given --cert/--key advertises no hash, so both client legs
        # verify the certificate: against relay_ca (mkcert, a private CA) or the
        # default trust stores. Never insecure=True.
        cert, key = _self_signed_cert(tmp_path)
>       with RelayProcess(cert=cert, key=key) as ready:

cert       = PosixPath('.../pytest-0/popen-gw2/test_external_relay_with_real_0/relay.pem')
key        = PosixPath('.../pytest-0/popen-gw2/test_external_relay_with_real_0/relay-key.pem')
tmp_path   = PosixPath('.../pytest-0/popen-gw2/test_external_relay_with_real_0')

.../web/relay_bridge/test_relay_bridge_e2e.py:706: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.../web/relay_bridge/relay_process.py:411: in __enter__
    return self.start()
        self       = <dimos.web.relay_bridge.relay_process.RelayProcess object at 0x7f80a12b17b0>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <dimos.web.relay_bridge.relay_process.RelayProcess object at 0x7f80a12b17b0>

    def start(self) -> RelayReadyInfo:
        deno = ensure_deno()
        web_dir = self._web_dir or find_web_dir()
        # No build here: start() must stay cheap (tests spawn many relays).
        # The build-if-needed step is ensure_web_dist(), called by the
        # RelayBridgeModule before spawning.
        cockpit_dir = self._cockpit_dir or find_cockpit_dist(web_dir)
        sdk_dir = self._sdk_dir or find_sdk_dist(web_dir)
        cmd = relay_run_cmd(
            deno,
            web_dir,
            "--port",
            str(self._port),
            "--host",
            self._host,
            cockpit_dir=cockpit_dir,
            sdk_dir=sdk_dir,
            serve_dir=self._serve_dir,
            cert=self._cert,
            key=self._key,
            auth_file=self._auth_file,
        )
        logger.info(f"starting relay: {' '.join(cmd)}")
        env = os.environ | {"NO_COLOR": "1"}
        self._process = subprocess.Popen(
            cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, env=env
        )
        assert self._process.stdout is not None and self._process.stderr is not None
        self._threads = [
            threading.Thread(target=self._read_stdout, args=(self._process.stdout,), daemon=True),
            threading.Thread(target=self._read_stderr, args=(self._process.stderr,), daemon=True),
        ]
        for thread in self._threads:
            thread.start()
        deadline = time.monotonic() + self._timeout
        while self.info is None:
            remaining = deadline - time.monotonic()
            if remaining <= 0:
                code = self._process.poll()
                self.stop()
                stderr = "\n".join(self._stderr_tail)
                state = f"exited with {code}" if code is not None else "still running"
                raise RuntimeError(
                    f"relay produced no ready line within {self._timeout} s ({state}); "
                    f"stderr tail:\n{stderr}"
                )
            try:
                self.info = self._ready_queue.get(timeout=min(0.1, remaining))
            except queue.Empty:
                code = self._process.poll()
                if code is None:
                    continue
                self.stop()
                stderr = "\n".join(self._stderr_tail)
>               raise RuntimeError(
                    f"relay exited with {code} before producing a ready line; "
                    f"stderr tail:\n{stderr}"
                ) from None
E               RuntimeError: relay exited with 0 before producing a ready line; stderr tail:

cmd        = ['.../hostedtoolcache/deno/2.6.10/x64/deno', 'run', '--frozen', '--node-modules-dir=none', '--allow-read=/home/runner...nner.../popen-gw2/test_external_relay_with_real_0/relay-key.pem', '--allow-net', ...]
cockpit_dir = None
code       = 0
deadline   = 507.507141382
deno       = '.../hostedtoolcache/deno/2.6.10/x64/deno'
env        = {'ACCEPT_EULA': 'Y', 'ACTIONS_ID_TOKEN_REQUEST_TOKEN': 'eyJhbGciOiJSUzI1NiIsImtpZCI6IjM4ODI2YjE3LTZhMzAtNWY5Yi1iMTY5LT...-version=2.0', 'ACTIONS_ORCHESTRATION_ID': 'd5787bb5-ba42-4e22-a734-f29d2b6ac439.tests._3_10_ubuntu-latest_false', ...}
remaining  = 19.899862111999994
sdk_dir    = None
self       = <dimos.web.relay_bridge.relay_process.RelayProcess object at 0x7f80a12b17b0>
stderr     = ''
thread     = <Thread(Thread-2672 (_read_stderr), stopped daemon 140186256111296)>
web_dir    = PosixPath('.../dimos/dimos/web')

.../web/relay_bridge/relay_process.py:375: RuntimeError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}