Audio tests related changes for kokoro and qwen3-asr - #4507
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
_analyze_audio() still computes RMS on empty input (producing warnings/NaN), which can make the new empty-audio validation noisy or flaky.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR updates the functional test framework around OpenAI-style audio endpoints to better support Kokoro TTS and Qwen3-ASR, including request parameter handling and stronger audio-output validation.
Changes:
- Extend OpenAI audio speech request params (voice + extra body) and adjust client-side voice passing behavior.
- Improve pytest parametrization to support tests requiring both a primary and auxiliary model on the same device.
- Add word-timestamp enablement plumbing for ASR nodes and harden audio validation for empty/invalid audio responses.
File summaries
| File | Description |
|---|---|
| tests/functional/utils/inference/serving/openai.py | Adds audio speech request params fields and adjusts transcription defaults. |
| tests/functional/utils/hooks.py | Refactors model parametrization to support auxiliary model selection per-device. |
| tests/functional/utils/generative_ai/validation_utils.py | Hardens audio metric computation and adds explicit empty-audio assertions. |
| tests/functional/utils/generative_ai/utils.py | Threads enable_word_timestamps through model/resource preparation. |
| tests/functional/object_model/python_custom_nodes/python_custom_nodes.py | Propagates enable_word_timestamps into calculator construction. |
| tests/functional/object_model/mediapipe_calculators.py | Emits enable_word_timestamps option in S2T node pbtxt. |
| tests/functional/object_model/inference_helpers.py | Uses OpenAI client sentinel handling for optional voice in TTS requests. |
| tests/functional/constants/ovms_messages.py | Updates expected voice-not-available message substring. |
| demos/common/export_models/export_model.py | Adds extra quantization CLI params to speech2text export path. |
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| print("Exporting speech2text model to ",destination_path) | ||
| if not os.path.isdir(destination_path) or args['overwrite_models']: | ||
| optimum_command = "optimum-cli export openvino --model {} --weight-format {} --trust-remote-code {}".format(source_model, precision, destination_path) | ||
| optimum_command = "optimum-cli export openvino --model {} --weight-format {} {} --trust-remote-code {}".format(source_model, precision, task_parameters['extra_quantization_params'], destination_path) |
There was a problem hiding this comment.
I guess we could use f-string type f"{}"
There was a problem hiding this comment.
I agree it would look better with f-strings. For now whole file uses older .format notation and requires refactor - I propose to move it to separate task.
There was a problem hiding this comment.
Let's keep it with .format(...) for now to stay in line with other parts.
See line 516-517:
optimum_command = "optimum-cli export openvino --model {} --task text-to-audio --weight-format {} {} --trust-remote-code {}".format(
source_model, precision, task_parameters['extra_quantization_params'], destination_path)
and many other places. If we switch to f-strings, we should do it everywhere across this script.
There was a problem hiding this comment.
🟡 Changes recommended
Audio parameters are dropped by the standard builder path, and streamed ASR responses fail when full-response mode is requested.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 10/10 changed files
- Comments generated: 2
- Review effort level: Balanced
//[VALIDATION_BRANCH=porlows1_kokoro_qwenasr]
25edf6e to
fcc02ec
Compare
//[VALIDATION_BRANCH=porlows1_kokoro_qwenasr]
🛠 Summary
Related to validation framework PR: https://github.com/intel-innersource/frameworks.ai.openvino.model-server.tests/pull/2790
https://jira.devtools.intel.com/browse/CVS-181965 Enable validation of Kokoro
https://jira.devtools.intel.com/browse/CVS-189463 voice field is not a string
https://jira.devtools.intel.com/browse/CVS-189724 Support Qwen3-ASR in OVM
🧪 Checklist
``