On-device Web Speech API

[evanbliu]

WebKittens

@eric-carlson

Title of the proposal

On-device Web Speech API

URL to the spec

WebAudio/web-speech-api#122

URL to the spec's repository

https://github.com/WebAudio/web-speech-api/

Issue Tracker URL

No response

Explainer URL

No response

TAG Design Review URL

No response

Mozilla standards-positions issue URL

mozilla/standards-positions#1157

WebKit Bugzilla URL

No response

Radar URL

No response

Description

The Web Speech API is a web standard API that allows developers to incorporate speech recognition and synthesis into their web pages. Currently, the Web Speech API spec does not mention where speech recognition occurs, and most implementers of the API perform speech recognition using a cloud service. Enabling on-device speech recognition support allows websites to ensure that neither audio nor transcribed speech are sent to a third-party service for processing.

[annevk]

Hey @evanbliu, we don't quite understand the design of this feature. At first glance it seems like it would be up to the web developer to decide how the speech recognition takes place, potentially even forbidding a user agent from only offering on-device recognition?

We also don't understand this requirement (also, UI requirements cannot be "must"):

User agents must obtain explicit and informed user consent before installing on-device speech recognition languages that differ from the user's preferred language or when the user is not connected to an Ethernet or Wi-Fi network.

Overall it seems like this feature might unintentionally reduce the amount of privacy.

[evanbliu]

Hi @annevk, websites can choose where speech recognition takes place, but that doesn't necessarily forbid the user agent from allowing the user to express a preference as well. If a website chooses on-device only but a user agent allows the user to choose cloud only, then the user agent would throw an error.

The spec language around this requirement was changed to: "The user agent can prompt the user for explicit permission to download the on-device speech recognition language pack." The purpose of prompting users for permission in certain cases is to mitigate the risk of abuse both in terms of bandwidth/storage consumption as well as fingerprinting.

[rniwa]

Hi @annevk, websites can choose where speech recognition takes place, but that doesn't necessarily forbid the user agent from allowing the user to express a preference as well. If a website chooses on-device only but a user agent allows the user to choose cloud only, then the user agent would throw an error.

That sounds like a new tracking vector? A website can tell whether the user has explicitly allowed on-device only vs. cloud-based by always asking on-device or vice versa for example.

[annevk]

Right, but also

If a website chooses on-device only but a user agent allows the user to choose cloud only, then the user agent would throw an error.

is a very good example of going against user agency. So I'm rather surprised it's cited as an example in favor of it.

[evanbliu]

That was simply an example of how a user agent might reconcile a discrepancy between the user's and the website's expressed preferences. In the case where a user only wants on-device speech recognition and the website requires cloud speech recognitions, there could be use cases where it may be better for the UA to throw an exception rather than to always proceed with on-device speech recognition.

The spec doesn't dictate how this interaction should behave, but it affords enough flexibility for user agents to add such options if needed. As of now, neither Chrome nor Firefox plan on providing users with an option to specify where speech recognition must occur.

[annevk]

Even if it was an example, it's not the outcome you would want. It seems reasonable for websites to express a preference, but ultimately the end user should be in control and the website should have limited-to-no visibility into what the end user has chosen.

[evanbliu]

I agree that the end user should ultimately be in control here, but there are use cases in which a user might want let the website know their preference. For example, a video conferencing website might run concurrent resource-intensive tasks including one or two HD video encoders (user video and screen shared content), multiple video decoders, audio encoders and decoders, background segmentation and replacement, possibly audio denoising, studio effects, etc. and thus prefer cloud speech recognition. If a user prefers on-device speech recognition, it could be beneficial to allow the website to disable studio effects, for example, to provide an optimal user experience.

As mentioned, there are certainly tradeoffs here with potential privacy implications here so I'm not necessarily advocating for this approach. If Safari wants to provide users with the ability to choose on-device vs. cloud speech recognition and always go with that then that would be fine too.

[szewai]

I agree that the end user should ultimately be in control here, but there are use cases in which a user might want let the website know their preference. For example, a video conferencing website might run concurrent resource-intensive tasks including one or two HD video encoders (user video and screen shared content), multiple video decoders, audio encoders and decoders, background segmentation and replacement, possibly audio denoising, studio effects, etc. and thus prefer cloud speech recognition. If a user prefers on-device speech recognition, it could be beneficial to allow the website to disable studio effects, for example, to provide an optimal user experience.

Is this a real request from some website, or we just assume there could be such case? I am not sure on-device recognition will always be resource-intensive (to an extent that affects performance), and I am not sure the website is in a good position to decide whether user's device has enough resources (e.g. device memory) to perform on-device recognition, so the "cloud-only" mode does not seem useful in general (and I don't know any site wants this).

Based on goals in the explainer, it seems mode attribute can have only two values: "on-device-only" and "default", where "default" is implementation defined: it can be on-device, or cloud-based or any of these two based on device conditions.

There also seems to be privacy concerns on availableOnDevice and installOnDevice. Before them, there is no way for website to know what languages are supported unless they start a recognition (user will notice from microphone prompt / UI). Now they can just query available languages? I don't see mitigations mentioned in the explainer.