Skip to content

Audio tests related changes for kokoro and qwen3-asr - #4507

Open
porlows1 wants to merge 7 commits into
mainfrom
porlows1_audio_tests_update
Open

Audio tests related changes for kokoro and qwen3-asr#4507
porlows1 wants to merge 7 commits into
mainfrom
porlows1_audio_tests_update

Conversation

@porlows1

@porlows1 porlows1 commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

🛠 Summary

Related to validation framework PR: https://github.com/intel-innersource/frameworks.ai.openvino.model-server.tests/pull/2790

https://jira.devtools.intel.com/browse/CVS-181965 Enable validation of Kokoro
https://jira.devtools.intel.com/browse/CVS-189463 voice field is not a string
https://jira.devtools.intel.com/browse/CVS-189724 Support Qwen3-ASR in OVM

🧪 Checklist

  • Unit tests added.
  • The documentation updated.
  • Change follows security best practices.
    ``

Copilot AI lite review requested due to automatic review settings September 3, 2026 12:22

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

_analyze_audio() still computes RMS on empty input (producing warnings/NaN), which can make the new empty-audio validation noisy or flaky.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR updates the functional test framework around OpenAI-style audio endpoints to better support Kokoro TTS and Qwen3-ASR, including request parameter handling and stronger audio-output validation.

Changes:

  • Extend OpenAI audio speech request params (voice + extra body) and adjust client-side voice passing behavior.
  • Improve pytest parametrization to support tests requiring both a primary and auxiliary model on the same device.
  • Add word-timestamp enablement plumbing for ASR nodes and harden audio validation for empty/invalid audio responses.
File summaries
File Description
tests/functional/utils/inference/serving/openai.py Adds audio speech request params fields and adjusts transcription defaults.
tests/functional/utils/hooks.py Refactors model parametrization to support auxiliary model selection per-device.
tests/functional/utils/generative_ai/validation_utils.py Hardens audio metric computation and adds explicit empty-audio assertions.
tests/functional/utils/generative_ai/utils.py Threads enable_word_timestamps through model/resource preparation.
tests/functional/object_model/python_custom_nodes/python_custom_nodes.py Propagates enable_word_timestamps into calculator construction.
tests/functional/object_model/mediapipe_calculators.py Emits enable_word_timestamps option in S2T node pbtxt.
tests/functional/object_model/inference_helpers.py Uses OpenAI client sentinel handling for optional voice in TTS requests.
tests/functional/constants/ovms_messages.py Updates expected voice-not-available message substring.
demos/common/export_models/export_model.py Adds extra quantization CLI params to speech2text export path.
Review details
  • Files reviewed: 9/9 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/functional/utils/generative_ai/validation_utils.py Outdated
Comment thread tests/functional/utils/inference/serving/openai.py
print("Exporting speech2text model to ",destination_path)
if not os.path.isdir(destination_path) or args['overwrite_models']:
optimum_command = "optimum-cli export openvino --model {} --weight-format {} --trust-remote-code {}".format(source_model, precision, destination_path)
optimum_command = "optimum-cli export openvino --model {} --weight-format {} {} --trust-remote-code {}".format(source_model, precision, task_parameters['extra_quantization_params'], destination_path)

@pgladkows pgladkows Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we could use f-string type f"{}"

@porlows1 porlows1 Sep 4, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it would look better with f-strings. For now whole file uses older .format notation and requires refactor - I propose to move it to separate task.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's keep it with .format(...) for now to stay in line with other parts.
See line 516-517:

optimum_command = "optimum-cli export openvino --model {} --task text-to-audio --weight-format {} {} --trust-remote-code {}".format(
                source_model, precision, task_parameters['extra_quantization_params'], destination_path)

and many other places. If we switch to f-strings, we should do it everywhere across this script.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Audio parameters are dropped by the standard builder path, and streamed ASR responses fail when full-response mode is requested.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 10/10 changed files
  • Comments generated: 2
  • Review effort level: Balanced

Comment thread tests/functional/object_model/inference_helpers.py
Comment thread tests/functional/utils/inference/serving/openai.py
@porlows1
porlows1 force-pushed the porlows1_audio_tests_update branch from 25edf6e to fcc02ec Compare September 4, 2026 14:11
//[VALIDATION_BRANCH=porlows1_kokoro_qwenasr]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants