#962: Captured Surface Control

Visit on Github.

Opened Jun 4, 2024

Uryyb GNT! V nz n ohqqvat pelcgbtencul rkcreg.

I'm requesting a TAG review of Captured Surface Control.

Summary

We introduce a new Web API that allows Web applications to:

  1. Read and write the zoom level of a captured display surface (tab or window).
  2. Produce wheel events in a captured tab or window.

Details

Further details:

  • I have reviewed the TAG's Web Platform Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): Screen Capture Community Group and WebRTC Working Group
  • The group where standardization of this work is intended to be done: WebRTC Working Group
  • Existing major pieces of multi-implementer review or discussion of this design: https://www.w3.org/2024/05/21-webrtc-minutes.html
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Google

Discussions

2024-08-26

Minutes

Martin: a bit of dispute... an issue raised... https://github.com/w3c/mediacapture-handle/issues/11

... 2 concerns - one is that this invents a new cross-origin communication mechanism. Maybe we should be using post message. Other concern: when you capture something and screen share it, the messages that come through from the other end.. e.g. requesting that page scroll or zoom - the design seems to make this indistinguishable from a regular input. Makes a machine available for remote control. This is at tension with user agency. It should be possible without the knowledge of the application, but heightened security...

Dan: abuse cases... we need to take seriously.

Martin: ... I'll take it, but we also should say "please talk to the wg"...

2024-09-09

Minutes

[We sent them a question that hasn't been answered yet, so skipping this week.]

2024-10-07

Minutes

Waiting on proponents.

2024-10-14

Minutes

Still waiting on proponents to respond/etc. Time to ask monseigneur Yasskin to apply the obligatory amount of force to colleagues.

Dan: pings Jeffrey to ping colleagues let's revisit at the plenary.

2024-10-28

Minutes

Dan: We have a response from Elad

Martin: ... it's fine ... the discussion of alternatives I think we covered off well ... risk mitigation is a bit soft ... maybe should push more on that.

Dan: security & privacy section is small and does not talk about abuse cases

Matthew: looks like this would work OK if someone viewing the share is already zoomed in - but this is about controlling the remote viewport. So this could improve a11y in that sense. But... is there any provision for the people on the call being able to control?

Martin: unclear - I think it's only local playback. No protocol that exists ...

Matthew: it's just "you are the person sharing this tab and it allows you to control scolling or zoom from there"

Martin: yes...

Matthew: from a cursory look - looks fine - but in something like this that has profound UX implications we should be asking for an a11y considerations section in the explainer. But they need to demonstrate that they thought about it.

Dan: Should we recommend that they get an a11y review from APA?

Yves: also an attack surface analysis.

Matthew: maybe something that could be good for a11y.

Dan: a11y checklist

Martin: ... not sure it's enough

Matthew: if you're sharing a web site...

Martin: ways in which different browsers render things differently...

Matthew: in some cases like collaborative editing ...

Martin: a lot you miss with the screen shot... in some cases that's a feature.

Matthew: re: checklist - APA does theirs as a questionnaire that allows you to open an issue somewhere else - whereas ours opens an issue in our repo.

Thank you for your reply and all the info in the Explainer. We discussed this on our breakout today. We still feel the explainer needs more information on possible abuse cases and a bit more discussion of attack surface. The security considerations talks about potential confusion, but doesn't talk about how the API could be abused by bad actors. So we recommend a security analysis (and there is a W3C process spinning up for this) but in the mean time if you could bolster the current security considerations doc with some discussion of abuse cases and mitigations that would great. As there's a lot going on UI-wise here, we'd really like to see an 'Accessibility considerations' section in the Explainer (it's totally fine to use this section to show what the positives are) - please could you add one? Please also consider requesting a review from the APA WG: https://github.com/w3c/a11y-request/issues/new/choose

we agree to post this

2024-11-04

Minutes

Jeffrey: I think this is probably ready to close as either satisfied or satisfied-with-concerns, but it's assigned to Martin so should probably wait until he's back.

Peter: Assigning it to the Nov 18 milestone.

Matthew: I'll also reply to the accessibility question.