Note Blink API owners are actively considering (and will likely soon approve) an I2S on this.

An explainer is under review here

We discussed this in a breakout today:

We think it's a useful advance to help websites recognize a user's speech on their local device, in line with our data minimization guidance. We see in the I2S thread that the CG is actively iterating on the shape of this API, and we'd like to let them come to some conclusions before we review the result. We've opened a design principles issue on the architectural issues around downloading large data files to support browser APIs, which apply across this and the Translation and Writing Assistance APIs.

We'll close this review for now, but please comment when the WG is closer to consensus or has architectural questions about the API.

The Audio WG has reached a consensus on the shape on the on-device speech recognition part of the Web Speech API spec. Please let me know if TAG would like to re-review this.

Could you update the explainer to describe the new consensus shape for the API?

Oops, I just sent out this PR updating the explainer: https://github.com/WebAudio/web-speech-api/pull/149

We discussed this in a breakout today, and we're enthusiastic about helping websites recognize a user's speech on their local device, in line with our data minimization guidance. Thank you for working on that.

With that said, we'd like to request several clarifications and API improvements:

Restricting recognition location

We see sites' desire to guarantee that speech isn't sent outside of the local device (for example to a cloud speech recognizer operated by the browser), especially for end-to-end-encrypted (E2EE) services. This is hard to actually guarantee: a valid browser architecture, especially for low-power devices, is to run the entire browser in the cloud and stream its UI down to the user. Similarly, a UA running on a low-power devices might prefer to run recognition in the cloud, perhaps to improve recognition quality. With those caveats, some of the TAG supports giving the site a way to encourage the browser to keep the speech local. Or, perhaps equivalently, for the site to tell the UA that it's going to keep the audio constrained to as few devices as possible, as a hint that the UA should do the same. This should not be a strong UA requirement, which means that it should not be called ondevice-only.

We don't see a reason to let sites require the recognition to happen in the cloud if the user prefers it to happen locally. Forcing a user to send unnecessary data to their UA's servers would violate our data minimization principles. So we recommend that recognition.mode==cloud-only" not exist. Perhaps just recognition.localOnly=true or recognition.onlyOnDevice=true, or have sites check the return value of the requestLocalRecognition([lang1, lang2, ...]) function suggested below?

We also note that there's an existing SpeechSynthesis API that mentions localService for voices. Could you add an analysis of how you're intentionally matching or diverging from that design?

Recognizing other users' speech

It's not clear whether it makes sense to have the browser download language packs to help websites recognize speech that comes from sources other than the current user speaking. This hesitance comes from two angles:

1. It seems more efficient to recognize speech once, and share the recognized text to all recipients, rather than having each recipient redundantly recognize it for themself. (https://w3c.github.io/sustainableweb-wsg/#success-criterion-client-vs-server-human-testable)
2. The download costs users something, and managing that makes the API significantly more complicated.

We could use a better description of the use cases that need these costs and complexity before we'll be comfortable endorsing that capability.

If this capability is justified, a site should request downloads of all of the needed languages in a single permission prompt. Without that, the user might misunderstand how much data they need to download before it would benefit them, which compromises their ability to consent. So, if this capability is needed, we suggest an API more like SpeechRecognition.requestLocalRecognition([lang1, lang2, ...]).

Personalization

In cases that are only recognizing the primary user's speech, we'd like the specification to analyze when it's safe and useful to personalize the speech recognition to that user. This could be as simple as "UAs MUST NOT personalize speech recognition", but we think there might be some utility in letting the start() method that doesn't take a MediaStreamTrack be personalized in order to better recognize the user's speech. For fingerprinting and other privacy reasons, we think personalization is only feasible if the site can't get both the audio and the personalized recognized text, and there might be other risks or problems we haven't thought of.

Fingerprinting

We note that availableOnDevice() has some inherent fingerprinting risks, similar to those created by permissions.query(). The explainer currently just says this lets sites "determine whether to enable on-device features or fall back to cloud-based recognition", which would be enabled just as well by a better return value for the install function, probably getting inspiration from the Translator.create() interface (cc/ @domenic). What specific site UI needs this sort of no-UI query function?

We noticed an additional fingerprinting risk, that the exact version of the downloaded language pack is likely detectable and likely to skew compared to the browser's major version. The specification should identify that risk and suggest or mandate ways to mitigate it. For example, to remove the risk entirely, you could require that each browser major version can only use 1 version of each language pack and that the packs are deleted when the user upgrades to an incompatible browser version. To only mitigate it, you could remove the start(MediaStreamTrack) overload or require packs be deleted when storage is cleared.

UI and user choices

The explainer says "user agents must obtain explicit and informed user consent before installing", but that doesn't appear in the specification, and the algorithm for installOnDevice() just says the UA "can prompt the user". This seems insufficient.

While the old version of this API exists, it would be good for browsers supporting it to help their users get local recognition even on sites that haven't adopted the new options. Could the spec suggest this? Would an omnibox icon be an appropriate UI in your Chromium implementation?

Similarly, there might be cases where a user actively wants the recognition to happen in the cloud, for example if their battery is unusually low, they want improved cloud-based recognition, or (speculating) to use a particular cloud service if there are several options with privacy tradeoffs. The same UI might be appropriate for giving them that control.

Nits

• In comparing this API to the proposed Translation API, we noticed that the proposed "download it" function here is missing an AbortSignal parameter to let the site abort the download.

Restricting recognition location

I really appreciate the TAG's nuanced perspective on this. (This is an area we're keeping an eye on for other APIs as well: https://github.com/webmachinelearning/writing-assistance-apis/issues/38.)

One technology that comes to mind is privacy-preserving cloud implementations. I believe we've seen some of these deployed in the private advertising spaces, and you could imagine them being deployed for these sorts of technologies as well.

I don't know the full details of how such technologies would be used here. E.g., is the privacy technologically guaranteed through mechanisms like TPMs, homomorphic encryption, multiple parties in the chain each of which only sees some of the data, etc.? Or the privacy mostly contractual, where the cloud model provider has some strong guarantee that they never look at the data? Do web developers and users have different feelings about these two levels of privacy guarantee?

My immediate concern is to avoid designing APIs which are future-incompatible with such technology. For example, consider a future where we designed an API of the shape { onDeviceOnly: true }, which when used means that the API only works on ~20% of devices (those with enough GPU memory). But then, one browser believes they have a strong-enough private cloud implementation, which would allow device coverage to go up to 100%. Are all sites that have chosen { onDeviceOnly: true } forever locked out of that 100% device coverage? Or do we break the meaning of { onDeviceOnly: true } to also allow private clouds? Neither alternative seems good.

I don't know what the best solution here is, but I think @jyasskin's suggestions of

With those caveats, some of the TAG supports giving the site a way to encourage the browser to keep the speech local. Or, perhaps equivalently, for the site to tell the UA that it's going to keep the audio constrained to as few devices as possible, as a hint that the UA should do the same. This should not be a strong UA requirement, which means that it should not be called ondevice-only.

seem promising.

Just noting that even though I posted the comment, its drafting was a team effort. In the suggestion that @domenic praised, at least @martinthomson and @christianliebel were essential to including the note that "keep it local" should just be a hint.

We appreciate the TAG's thoughtful and thorough feedback on our proposal to add on-device speech recognition support to the Web Speech API. We discussed this feedback at the monthly Audio Working Group meeting with representatives from Google and Mozilla. Below, we address each of the major concerns raised:

1. Restricting Recognition Location

We accept TAG's recommendation to remove the cloud-only option from the API. This aligns with the current direction of implementation across browsers—Firefox has no plans to support cloud-only recognition, and Chrome is also moving away from it. That said, we acknowledge that cloud-based speech recognition may still be preferred in certain situations, such as:

• On low-power devices that lack sufficient compute resources.

• When resource-intensive on-device features are already in use.

• When cloud recognition offers better quality in specific contexts.

Importantly, we must also support use cases where audio must not be sent to third-party services, such as for regulatory or compliance reasons. While we acknowledge that confidential computing may eventually offer a viable solution for such cases, we do not intend to support that path at this time. Instead, we aim to design the API to be extensible, so that support for confidential computing can be added later if needed.

To accommodate current needs, user agents that cannot guarantee local-only processing may throw an error, allowing websites to make informed decisions based on the level of assurance required.

2. Recognizing Other Users' Speech

We recognize the efficiency argument in centralized recognition (e.g., WebRTC). However, we believe MediaStreamTrack support enables flexibility, allowing:

• Sender-side recognition, where the originator of speech recognizes and distributes captions.

• Receiver-side recognition

We're open to extending the installation API to support multiple language packs in a single call (e.g., installOnDevice([lang1, lang2, ...])), improving both user experience and consent clarity. We note, however, that the current API returns a Promise<boolean> indicating installation success. Supporting multiple languages could introduce ambiguity if one language fails. We will explore a more expressive return format to address this.

3. Personalization

We accept the TAG's recommendation and will add language to the specification stating that:

User agents MUST NOT personalize speech recognition.

We agree that this approach avoids fingerprinting and privacy risks while keeping the API simpler and more secure.

4. Fingerprinting

We acknowledge the fingerprinting risks posed by functions like availableOnDevice(). To mitigate these risks:

• We will align with the Web Translation API’s privacy-preserving approach.

• Both Chrome and Firefox will support only one language pack per language at a time, reducing the variability that could lead to fingerprinting.

• Language packs will be cleared when browser storage is cleared, ensuring consistent behavior with other privacy controls.

For Chrome specifically, we will implement the same fingerprinting mitigations used in the Web Translation API. These are detailed in the following document: Fingerprinting Mitigations.

5. UI and User Choice

We acknowledge that some users may prefer cloud-based recognition in certain scenarios—for example, to conserve battery or access higher-quality recognition. In principle, we support giving users control over this choice through appropriate UA-level mechanisms.

However, neither Chrome nor Firefox currently plan to expose explicit UI-level controls for this. Firefox is committed to supporting only on-device speech recognition. Chrome is also moving away from cloud-based recognition and plans to phase it out over time.

The specification will be updated to use the phrasing "may prompt the user" to allow user agents the flexibility to implement privacy-preserving countermeasures in a way that best fits their platform and user base, without prescribing a specific UI. For instance, Chrome will rely on non-UI mechanisms to mitigate fingerprinting risks, rather than requiring explicit user prompts.

6. Nits

We are open to adding an AbortSignal parameter to the download/install function, consistent with modern web platform design.

We note that for Chrome's implementation of the Web Translation API, aborting the request does not cancel the actual download for privacy preserving reasons, but rather stops associated download progress events.

Thanks for the reply and initial changes! Here are some initial thoughts from my perspective. These haven't been vetted by the whole TAG yet:

1. Restricting Recognition Location

A cloud server that the UA uses for recognition would be a second-party service, since the user is the second party, and the server operates on their behalf. I still support a hint that the site is acting to reduce the number of machines that could see the audio, which will encourage UAs to do the same, but I don't think it can honestly be called "ondevice-only" and still give UAs the flexibility they need to act on their users' behalf. I recognize that Chrome and Firefox don't have any plans to let users offload this work to cloud services, but we should design the API to accommodate future UAs that might want to explore that direction.

I would support a statement that, at least given this hint, UAs MUST NOT expose the audio to any third parties, in case that helps these concerned partners be more comfortable with the change.

2. Recognizing Other Users' Speech

It's true that the current design enables flexibility, but it's not clear that this is user-serving flexibility. It might be, but I don't see use cases described in the explainer that show that users need the flexibility. "scenarios where personalized or local processing is preferred" should instead describe a few of those scenarios. Note that https://github.com/WebAudio/web-speech-api/pull/150/files specifically bans personalized processing, and local processing is just as possible with sender-side recognition, which reinforces the possibility that there are no such scenarios.

4. Fingerprinting

Note that the Fingerprinting Mitigations document isn't world-readable, so most of the TAG can't see it.

I generally like the direction of the Web Translation API’s approach to privacy, but the TAG hasn't fully analyzed it, so I don't want to say it's definitely enough. I think a list of use cases for the availableOnDevice() query would help the rest of the TAG get more comfortable with the idea.

Does "support only one language pack per language at a time" mean that each browser major version will be pinned to exactly one pack version per language? That's tighter than https://github.com/webmachinelearning/writing-assistance-apis/pull/47 was willing to require, but it solves the extra fingerprinting problem.

Thanks for the notes, Jeffrey! I haven't had a chance to meet with the Audio WG yet, but here's my attempt at clarifying some of your questions.

1. Restricting Recognition Location

The recognition.mode = "ondevice-preferred" option is indeed intended to serve as the "hint" that websites can use to express a preference for on-device speech recognition, without the UA guaranteeing it. This allows UAs flexibility, for instance, to use their own second-party cloud services if it benefits the user (e.g., on low-power devices) and the site has only indicated a preference.

However, we maintain the critical need for a mechanism that allows websites to guarantee that audio is not sent to any external service—be it a second-party or third-party service—for processing. The recognition.mode = "ondevice-only" option is designed to serve this specific requirement. If a UA cannot process the audio solely on the device (without any external network transmission of the speech data for recognition purposes), it should then throw an error when this mode is requested. This provides a clear assurance for applications with strict data residency or privacy requirements.

2. Recognizing Other Users' Speech

The mention of MediaStreamTrack support was in response to the discussion on efficiency and sustainable speech recognition. It enables developers to implement either sender-side or receiver-side captioning. The previous note on personalization was incorrect and has been removed—while local speech recognition can support personalized sender-side captioning when audio is captured via microphone, it does not apply to receiver-side captioning using a MediaStreamTrack.

4. Fingerprinting

Apologies for the inaccessibility for the document--we're unable to share the exact contents of the document at this time, but the PR you linked to provides a thorough explanation of the privacy preserving countermeasures of the Web Translation API that we're planning on adopting here.

Here are some use cases that availableOnDevice() aims to address:

• Conditionally Offer Features: Decide whether to present UI elements or enable features that rely on on-device recognition before prompting for installation. This avoids offering a feature that isn't viable.
• Graceful Degradation/Enhancement: Allow the site to immediately understand if on-device is an option. If not, it can fall back to alternative mechanisms (like cloud-based services it operates, or informing the user about limitations).
• Resource-Informed UI: For example, a web application might choose to display a "Transcribe Locally" button only if availableOnDevice() indicates potential support.

Thanks; I think we have almost enough to re-discuss this. My last remaining questions are:

1. What's one concrete website that might adopt this API only if it provides the mechanism to guarantee that audio is not sent across the network, even encrypted to another device controlled by the user? Have they described their requirement in public in a place the TAG could read?
2. What's one website that needs to recognize audio on the receiver side, and can you point us to a place where they've documented how this architecture is better for their users than either having senders recognize the audio or having the cloud server recognize it?

It would be great if the agreed changes and the concrete use cases for parts of the current design that the TAG (or others) were concerned, could be reflected in the explainer, asap.

Thanks.

Google Meet is one site that would only adopt this API if it provides a mechanism to guarantee that on-device speech recognition is used, though not for typical data residency reasons. Specifically for Google Meet, the performance of the latest on-device speech recognition language packs, in addition to biasing support, meets their stringent requirements, whereas the Open Speech API that powers the cloud implementation of the Web Speech API in Chrome does not. If on-device speech recognition is not available, Google Meet would continue to use its own cloud speech recognition implementation instead of the Web Speech API.

More commonly, developers have been requesting on-device speech recognition support simply because they do not want to send audio data to Google or other speech recognition services for various reasons. Here are some examples of these requests: https://webwewant.fyi/wants/55/ https://github.com/WebAudio/web-speech-api/issues/108 https://stackoverflow.com/questions/49473369/offline-speech-recognition-in-browser https://www.reddit.com/r/html5/comments/8jtv3u/offline_voice_recognition_without_the_webspeech/

padenot@mozilla.com may have other examples of EU partners with regulatory requirements regarding this.

As for receiver-side captioning in a WebRTC scenario, one primary benefit is that in a large meeting, if only one person requires captions, then only that person has to install and run on-device speech recognition. Otherwise, with sender-side captioning, every single person in the meeting would have to run on-device speech recognition. This document isn't viewable externally, but http://go/receiver-captions (Google only) has some trade-offs for receiver-side captioning specifically for Google Meet.

I'll work on updating the explainer in the next day or so with all of the changes and discussion in this thread!