#602: Early design review of the **updated** Multi-Screen Window Placement API

Visit on Github.

Opened Jan 26, 2021

HIQaH! QaH! TAG!

I'm requesting a TAG review of the updated Multi-Screen Window Placement API.

This proposal introduces incremental improvements to existing screen information and window placement APIs, allowing web applications to be intentional about the experiences they offer to users of multi-screen devices. The API shape has been updated since our last TAG review with feedback from partners and web platform API experts (example).

Further details:

We'd prefer the TAG provide feedback as (please delete all but the desired option): 💬 leave review feedback as a comment in this issue and @-notify @michaelwasserman

Thank you!

Discussions

Comment by @tomayac Jan 27, 2021 (See Github)

This looks like a copy & paste glitch. The changes are mentioned in CHANGES.md.

Comment by @michaelwasserman Jan 27, 2021 (See Github)

Thanks for catching that, @tomayac! I updated the initial comment above.

Discussed Mar 8, 2021 (See Github)

Ken: has encorporated feedback from me and Jake Archibald. so that's good. Seems sensible. One thing now is that when youc call getscreens you get a screens object ... could be a naming issue... screens.screens... An earlier version of this implementedis what is in the spec today. They are incorporating this into a new explainer - it's not spec'ed.

Dan: should we try to close this?

Ken: I'm concerned it might ship if we don't provide our feedback...

Ken: I think this is pretty good - they are going through and origin trial to get more feedback.

Dan: security and privacy? I can review.

Rossen: where does end up? what spec?

Dan: second screen working group ... is what's indicated. Is there multi stakeholder support?

Ken: will leave naming feedback.

Peter: somee thoughts - aware of work on foldable screens... Intending to harmonize. My concern is that someone ships part of this before rest is baked and we end up with multiple multi-screen APIs...

Ken: I think we're pretty aware... they want to make sure this fits together...

Peter: it's a meta-concern.

Peter: my other concern - takes all the screens and puts them into one virtual coodinate space - does that have the best developer ergonomics? Should we be able to target a specific screen object?

Ken: 3 TVs as a big canvas... [as a use case]

Peter: in some cases a developer will want to taget a window a specific screen - and with this approach you have to do the math to figure that out.

Rossen: in the other case where we consider screen-local coordinates - you still have to do the math... however you look at it you have to do math.

Ken: unless you want to do something on a specific screen full screen, which could be a common use case.

Rossen: requestfullscreen...

Peter: but you're not always gonna want full screen. Just a question of convenience for developers.

Rossen: i'd like to see how other platforms are handling this...

Dan: worth asking the question on the issue...

Peter: will do.

Peter: changing behaviour of how it works today.. breaking exisitng implementation

Peter: my proposal is to add an optional parameter - move to x,y (& this screen). And also wondering how moveto works today in a multi-screen setup.

Rossen: pretty sure you have a canvas (across) both windows... could be platform specific.

Rossen: in windows we draw the distinction between displays and screens... multiple displays that you can enumerate...

Dan: point on developer ergonomics is good. Web doesn't necessarily want to map platform features into identical js apis. Wants to make sense for web devs.

Comment by @kenchris Mar 8, 2021 (See Github)

I find Promise<Screens> getScreens(); a bit confusing due to .screens property. It can lead devs to write code like

screens.screens.length and even the examples use screensInterface.screens.length in order to avoid saying screens twice. Maybe we should just call it ScreenInfo or similar, like Promise<ScreenInfo> getScreens() - naming is hard.

I was also thinking about something like "ScreenDevices", but I still hope we can find a better name

Comment by @plinss Mar 8, 2021 (See Github)

I'm wondering if it would help developer ergonomics to also allow an optional screen argument to the Window.moveTo() method, which if passed, makes the x, y coordinates relative to the passed screen. If added, a similar mechanism should be added to Window.open() (perhaps changing the features argument from a string to an object).

Discussed May 1, 2021 (See Github)

Ken: ... some changes with this....

[reviewing]

Dan: there's lots of fingerprinting info exposed as document in their answers to the s&p questionnaire. Is any of this mitigated against e.g. via a permission request? this would seem to indicate that access to this kind of data is limited behind a permission.

Comment by @quisquous May 3, 2021 (See Github)

I find Promise<Screens> getScreens(); a bit confusing due to .screens property.

We have that filed on https://github.com/webscreens/window-placement/issues/50 as a place to discuss. As you say, naming is really hard. I don't think anybody has come up with a better name yet and so we've been punting on that so far.

I'm wondering if it would help developer ergonomics to also allow an optional screen argument to the Window.moveTo() method, which if passed, makes the x, y coordinates relative to the passed screen. If added, a similar mechanism should be added to Window.open() (perhaps changing the features argument from a string to an object).

I worry that it might be confusing to have the coordinates change their semantics if an optional third parameter is present. I wonder if it might be better to pick one of (a) x and y are relative with optional screen or (b) x and y are absolute, no screen parameter.

Comment by @torgo May 12, 2021 (See Github)

@michaelwasserman Just reviewing this in the context of our virtual f2f. Appreciate what you've written here about fingerprinting mitigation.

Comment by @michaelwasserman Jul 10, 2021 (See Github)

Thanks for the comments and feedback, @kenchris, @plinss, and @torgo. Thanks for the reply, @quisquous.

@kenchris: we're open to naming feedback on webscreens/window-placement#50 and I added a comment there, but we haven't necessarily found a more compelling pattern than our current proposal.

@plinss: I hope we'll continue improving the state of window placement on the web with future proposals, perhaps with something akin to your suggestion, and nothing in this proposal precludes such future work. Perhaps developer ergonomics would be served best by moving away from disparate moveTo(), resizeTo(), etc. functions, and adding a new function that can set a bounds rectangle, and potentially related window placement state (e.g. maximized/zoomed, left-/right-snapped, etc.) in a single request. That could support either global cross-screen coordinates and/or screen-relative coordinates (with an optional screen parameter, or a default assumption of relative placement within the current screen). There's a lot to consider, and this proposal aims to start small, but I have some thoughts in the repo's additional_explorations.md and aim to continue exploring.

Is there any other feedback from TAG members that we can consider during this early design review? Thanks again!

Discussed Aug 30, 2021 (See Github)

Dan: I think this is ready for closure based on the feedback we got on the 10th, unless there's additional feedback we want to give. I did put the multi stakeholder thing in here. We could say it remains an issue/question.

Rossen: added some docs to issues around naming and use cases... there's quite a bit of content... looks like the issue they link to - https://github.com/webscreens/window-placement/issues/50 trying to figure out if they're ok with our proposed renaming. The current name is window.getscreens.screens... we flagged. I don't see that they are ready to change it....

Peter: new explainer says getscreens is directly on window.

Dan: have they taken our feedback on board and are debating it, or have not really understood our feedback?

Peter's cat: MAAAAW

Rossen: it seems like they are sympathetic to the proposal.

Dan: could be an argument for closing

Rossen: let me touch base with Ken first. I remember a long discussion about this one with Peter and we went through all of the apis

Dan: plenary, proposed closed

Discussed Sep 1, 2021 (See Github)

Rossen: wrote closing comment

Comment by @atanassov Sep 15, 2021 (See Github)

@kenchris and I did another pass at this review during our Gethen vf2f. Thank you for addressing our earlier feedback about API naming and developer ergonomics. In particular we prefer the getScreenDetails name over getScreens especially in combination of getScreenDetails().screens. Further, the detailed exploration document you linked to goes into great details on how to make the API better and since this is an early design review we hope this will be taken into consideration for the final API shape.

At this point we are happy to close the review as satisfied and look forward to the progress you'll make. Thank you and good luck.

Comment by @michaelwasserman May 9, 2022 (See Github)

Greetings TAG!

I am requesting an supplemental early design review of a proposed API enhancement.

The proposed enhancement would allow sites with the window-placement permission to initiate a multi-screen experience from a single user activation. A partner requested this functionality, when integrating the Multi-Screen Window Placement API in their web application.

We'd prefer the TAG provide feedback as: comment in this issue and @-notify @michaelwasserman

Thank you! Mike Wasserman

Discussed Jun 13, 2022 (See Github)

Peter: ..

Dan: popping up multiple windows can be a way that bad actors try to confuse the user by ocluding bits of browser ui, making it hard for them to get back to what they were doing... but is this only in a multi screen scenario..?

Peter: when you have window placement permission already. And you've got an activation so you can place fullscreen content. And what youc an now do is let you place popups on other screens. So you won't get 100 popoups all of a sudden, but you could go to fullscreen mode and get popups on other screens. That may be what you want in some scenarios, but share the concern.. can it be abused by.. not always fullscreen on the most ethical sites.

Dan: I'd like to unerstand if they';ve really thought about the abuse scenarios

Peter: they list mitgations in the explainer

Dan: are they considering browsers on keyboardless devices? They talk about esc key for instance. Not available to everybody. Are we clutching at straws? Minor change.

Dan: minor change.. should we just close?

Peter: I'm ok to close it I guess. I do have concerns about how badly it can be abused...

Dan: Proposed comment on multi-stakeholder:

@michaelwasserman when we closed this issue last year we indicated a concern related to multi-stakeholder. I note that the API is still being developed in the Second Screen Community group and Chrome Status does not list any other engine that has expressed interest. Has this moved in any way towards the Second Screen working group and has there been any interest expressed by other implementers?

Rossen: Looks different from the initial ... Now... The way I see this is that this is generally reducing friction.. not enabling something that is not possible. Question is: is that friction really that bad?

Peter: how can it be abused? To me the abuse could be "i am on e.g. instagram, i press full screen, i get ads popping up on all my other screens full-screen".

Rossen: If I have an app that is a fullscreen app and I launch another app that is also full screen the first app may or may not exit full screen. With multi-screen.. with modern window management capability - i can separate by sections... launch one of those apps and my screen layout will not change on the 2ndary screen unintentionally... I don't see where they address privacy & tracking... Taking over screens, especially multiple screens, could...

Dan: I don't think they specifically address that in mitigations. They're not talking about privacy and tracking

Rossen: trying to see if they cover it somewhere else

Dan: indicated they're interested in moving it to the second screen wg, so want to ask about that. Would make sense for someone else to ask about the abuse scenarios?

Rossen: I want to expand the comment about privacy and abuse. Does it need to be part of this review or fork off to a different review? it's very related but it's an extension.. why are they reopining the same issue

Peter: could have been a new review request

Rossen: if we ask them to open a separate issue they'll have to go through multistakeholder etc again

Peter: we dont' need to religitate everything we already reviewed. Fair to push them on multistakeholder because that's left over.

Dan: I can also post about abuse.. but I think Rossen had it more in his head

Comment by @torgo Jun 13, 2022 (See Github)

@michaelwasserman when we closed this issue last year we indicated a concern related to multi-stakeholder. I note that the API is still being developed in the Second Screen Community group and Chrome Status does not list any other engine that has expressed interest. Has this moved in any way towards the Second Screen working group and has there been any interest expressed by other implementers?

Comment by @atanassov Jun 13, 2022 (See Github)

One more question is about security and privacy - I can't find a relevant write up about it and was wondering if you're assumption is that the current questionnaire answers cover the newly added functionality too?

Comment by @michaelwasserman Jun 13, 2022 (See Github)

Thanks for these inquiries!

@torgo: The API was adopted as a Second Screen Working Group deliverable, see the latest charter and CfC to publish a FPWD. We are getting valuable feedback from Mozilla, which just continued at a recent SS WG vF2F, but have no explicit positive signals from another implementer.

@atanassov: The proposal includes mitigations to limit the potential for accidental misuse or abuse. The spec was recently updated per Mozilla's enhancement feedback, to clarify Security Considerations around deceptive cross-screen placement. The API's questionnaire generally covers this minor enhancement, but the Spec and Explainer more directly address the pertinent considerations of this change.

Discussed Jul 1, 2022 (See Github)

Dan: reviewing response on our concerns it seems to address our issues

Rossen: reviewing Mozilla's standards position... The issue with taking over a monitor that users doesn't pay attention to is not as fringe as it's being presented. Often, people use laptops with a large screen monitor (at the office for example) and aren't paying any attention to the main, laptop's screen.

Rossen: leaves comment

Comment by @atanassov Jul 26, 2022 (See Github)

Hi @michaelwasserman – @torgo and I are just reviewing this at our f2f. It looks like the issue raised by @annevk hasn't been adequately addressed. That seems to be a pretty common use case (where a malicious site might seek to take over a display and mimic the user's OS (or similar) in order to steal credentials. Also as we've been reviewing it seems to us like this is more than a small enhancement. This proposal extends several technologies that are not part of multiscreen: user activation, Element.requestFullScreen and window.open. Since the scope is a bit wider than a small enhancement on an existing API. Can we ask that you to fork the topic and file a new design review request for this?

Discussed Aug 29, 2022 (See Github)

Tess: prefer to close it when they actually file the follow-up.

Dan: added note to clarify status but left open

Comment by @torgo Aug 29, 2022 (See Github)

Just marking this as pending editor update for now - we will close when additional issues have been filed.

Comment by @michaelwasserman Aug 31, 2022 (See Github)

Thank you, @atanassov and @torgo! Please see the newly opened TAG review request and the Explainer's new Security Considerations, which documents and attempts to address the concerns raisied by @annevk; I also replied directly to that Mozilla standards-position thread. Thank you for your time and patience!

Discussed Nov 28, 2022 (See Github)

Dan: can this be closed? They've pointed us to a new review, #767

Peter: looking at the difference between the two... is 767 looking at only a specific part of this spec? They have a separate explainer.

Dan: last feedback from Rossen about specific issues. New review request was in response - attempts to address concerns. Either this is a replacement, or ..

Peter: we had closed it in 2021 and they reopened it to ask us to look at multi screen part. 602 changed.. I believe we can close 602 now and move on with 767

Comment by @torgo Nov 28, 2022 (See Github)

We're closing this as the review has moved to #767.