#856: API for capturing all screens

Visit on Github.

Opened Jun 12, 2023

こんにちは TAG-さん!

I'm requesting a TAG review of "capture all screens" (getAllScreensMedia).

In some scenarios (e.g. legal compliance or training purposes) it must be ensured that all screens attached to a client are captured. With getDisplayMedia, a web application would need to call the API N-times, asking the user to pick a different screen each time. This is a cumbersome and error-prone process and it can't be guaranteed that all the screens are captured. We propose the getAllScreensMedia API that allows capturing of all screens without user interaction. To protect the users privacy, the API is safe-guarded by an enterprise policy and the user must be notified before capture can happen and while capturing is happening.

  • Explainer: Url
  • Specification URL: Spec Url, Code
  • Tests: No WPT as a policy needs to be set. Other tests
  • User research: Developer feedback
  • Security and Privacy self-review: TBD (will be posted in a comment below)
  • GitHub repo: Url
  • Primary contacts (and their relationship to the specification):
    • Simon Hangl (@shangl, Google)
    • Elad Alon (@eladalon1983, Google)
  • Organization(s)/project(s) driving the specification: Google
  • Key pieces of existing multi-stakeholder review or discussion of this specification: Developer feedback, SCCG presentation, SCCG meeting minutes (SCCG)
  • External status/issue trackers for this specification (publicly visible, e.g. Chrome Status): Chrome status

Further details:

  • I have reviewed the TAG's Web Platform Design Principles
  • Relevant time constraints or deadlines: N/A
  • The group where the work on this specification is currently being done: SCCG
  • The group where standardization of this work is intended to be done: N/A
  • Major unresolved issues with or opposition to this specification: N/A
  • This work is being funded by: Google

We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify @shangl

Discussions

Discussed Jun 1, 2023 (See Github)

Dan: Do we need this in the web platform itself? isn't his better put at the system level? Why does this need to be in a browser API?

Sangwhan: Proctored exams (monitored exams), monitoring employee productivity, call centers...

Amy: it says it's behind an enterprise policy - we've pushed back before on this kind of surveillance in the web platform ... eg. managed device API

Hadley: aren't enterprise policies usually at the OS level?

Dan: we said fine if they want to add it to Chrome as their own thing, but maybe not appropriate for standardisation

Sangwhan: jist is that the proctored test is usually a web application - on the web. You'd either have to make a native application to do the same thing - which is what is going on right now - this moves some of that horrible stuff to the web. I can see who wants to use it and why it makes sense... You can give the user a capture that this is being captured... User permission requested. In the browser. In native code, all bets are off. In that sense I think it's better than being in native.

Amy: they don't list exams as one of their use cases...

Hadley: I'm hesitant to respond to a use case they haven't made explicit.

Yves: if it's an enterprise based installation, they can probably install a native extension to chrome that may be needed for the same thing. Dangerous to have that avalable for everyone on the web.

Amy: they don't go into detail about the end user needs, just high level summary use cases

Sangwhan to leave a comment

Comment by @cynthia Jun 20, 2023 (See Github)

Why does this need to be provided as a web-facing API? All three provided use-cases could likely be done with closed (e.g. extension specific) APIs and you could deploy that as an enterprise policy, which would alleviate the risks of abuese by having it exposed on the web.

Comment by @eladalon1983 Jun 20, 2023 (See Github)

Why does this need to be provided as a web-facing API? All three provided use-cases could likely be done with closed (e.g. extension specific) APIs and you could deploy that as an enterprise policy, which would alleviate the risks of abuese by having it exposed on the web.

An extension API requires the installation of an extension, which implicitly grants permission to the extension to access a myriad other APIs that the admin might not even know about, let alone wish to allow. In some regards the extension path is actually riskier than a Web-exposed path, where the default stance is more security-conscious, and the holes enterprise policies poke through the wall are more deliberate.

Discussed Jul 1, 2023 (See Github)

Amy: feedback before is that this is an enterprise thing ... maybe not in the web platform? Their response assumes an extension is untrustworthy, but I don't understand, as in this situation wouldn't they be the ones providing the extension?

Yves: if you know that the enterprise is putting extensions on you then you know what to expect - if it's part of the web platform in general then you don't.

Amy: ... APIs the admin might not know about.. isn't it the duty of the admin of an enterprise system to know about what they're telling their users to install/use?

Dan: ... safe to browse...

Dan: leaves comment

Discussed Jul 1, 2023 (See Github)

reviewing the comments from Elad

Amy: the fact that it's constrained by admin allowlist suggets it needn't be on the web

Dan: like managed device api

Sangwhan: if another implmeenter is interested, they could converge and figure it out and maybe there's a path to standardisation. But probably recommend we say it's not part of the web

Dan: we said unsatisfied on managed device because of the use cases - not in users best interests, contravenes best interests of the user. Reasonable for us to give the same view. Unsatisfied. Adding as a configuration option on enterprise deployments for chrome, that's their business.

Amy: +1

Dan: we can reinforce the user can't "just walk away", and making these policies harder to implement differentiates the web platform from other platforms close and comment

Comment by @torgo Jul 18, 2023 (See Github)

Hi @eladalon1983 I'm not sure we entirely follow your argument. As @cynthia said, there are risks for abuse by exposing this to the entire web. From a safe to browse point of view, it should be difficult for the user to get into a scenario where they are sharing their screens. A risk I see that is not discussed in the explainer or the spec is users' responses to permission prompts being gamed – e.g. by phishing attacks claiming to be customer service for a financial provider. At the very least, it feels like there needs to be more in the spec about the potential abuse scenarios and mitigations, beyond presence of a "privacy indicator".

Comment by @eladalon1983 Jul 18, 2023 (See Github)

A risk I see that is not discussed in the explainer or the spec is users' responses to permission prompts being gamed

@torgo, a user cannot chance upon an arbitrary malicious site and be shown a dialog to share all their screens with that site. This API is restricted to origins allowlisted by the admin. That configuration should even be enough to forego a user-prompt altogether, if the user was warned earlier that the device might be subject to surveillance at any time, allowing them to walk away from the device rather than use it if they don't accept that.

That said, I believe other concerns (XSS) have prompted discussion of potentially moving this API off of the Web. @shangl would know the status of these and whether this TAG review request is still relevant.

Comment by @torgo Aug 1, 2023 (See Github)

Hi @eladalon1983 we've just been discussing in our TAG f2f today. We're of the mind that, for many of the same reasons articulated in our review of the managed device API we don't think this should be part of the web platform. The kind of surveillance you're talking about (where, by the way, people might not be able to make the decision to "walk away" without also losing their job or incurring other harms) is not something we're comfortable signing off on.

Comment by @eladalon1983 Aug 16, 2023 (See Github)

The kind of surveillance you're talking about (where, by the way, people might not be able to make the decision to "walk away" without also losing their job or incurring other harms) is not something we're comfortable signing off on.

I understand the discomfort and even share it, especially after seeing some public backlash to other controversial APIs. So I'd like to mention[*] for record that this very unfortunate dilemma - "accept or quit" - is already present in the lives of countless people, whose employers provide a machine running Windows/Mac/Linux with a native application installed that surveils the employee. Such existing native applications can even surveil the user without their knowledge. The API proposed by Simon here would take steps to improve our world, because it would at least ensure that the user knows the machine is subject to potential surveillance.

Creating a world completely free of surveillance is beyond our powers; we take pride in the little strides forward we can make.

-- [*] Also mentioned here.

Discussed Oct 1, 2023 (See Github)

Amy: We proposed close ages ago - can we close it?

Dan: [closes]