#689: Viewport Segments Property

Visit on Github.

Opened Nov 9, 2021

Braw mornin' TAG!

I'm requesting a TAG review of the Viewport Segments Property after taking into account feedback (e.g. webscreens/window-segments#12) around integrating our original Window Segments API proposal (initial TAG review here: #492) with the Visual Viewport API.

Web developers targeting foldable devices want to be able to effectively lay out the content in a window that spans multiple displays. However, the web platform does not yet provide the necessary primitives for building layouts that are optimized for foldable experiences. The Viewport Segments Property will provide a list of logical segments of the viewport that web developers can use to target with different views, as appropriate based on size and configuration of those segments.

Further details:

  • I have reviewed the TAG's API Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
  • The group where standardization of this work is intended to be done ("unknown" if not known): Unknown
  • Existing major pieces of multi-stakeholder review or discussion of this design:
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as:

💬 leave review feedback as a comment in this issue and @-notify [github usernames]

Discussions

Comment by @kenchris Nov 11, 2021 (See Github)

I am very happy to see this being part of VisualViewport now. I like the figures/images from the explainer and think it would be nice to add those to the spec as well

Comment by @JensenPaul Nov 16, 2021 (See Github)

It seems like this API might expose hardware characteristics to the web.  Would it make sense to add text to the explainer's "Security & Privacy" section discussing how identifying these characteristics are, quantifying the fingerprinting risks, whether they're exposed by other APIs, and what mitigations you've considered?

Comment by @scottlow Nov 19, 2021 (See Github)

@kenchris thanks! I can take an action item to get those added.

@JensenPaul, we've called out the additional bits of entropy and the fact that for some devices they're already exposed via window.screen properties in the questionnaire. The "Security and Privacy" sections of each explainer (I figured I'd comment here to address your comment in #690 as well) detail the mitigations we've put in place for each new primitive.

Comment by @kenchris Dec 7, 2021 (See Github)

On devices with an actual foldable display, it makes a lot of sense to talk about segments when folded, with the addition that the segment in the middle (the actual folded part) should also be exposed as it might not be touchable (or touch events might not work well). @darktears @diekus @lauramorinigo

The reason I am asking is that it would be nice to discuss and show example of this in the explainer/spec. Like I don't know what the current thinking is for foldable displays. Will it just expose 3 regions, with no overlap (first, fold, last), or will the fold area overlap with the first and last region? Would all of this work with this API?

Some gamer laptops have 1.5 screen, basically a mini display above the keyboard that can be "lifted up" together with the main display to become a larger "foldable" monitor. You could imagine the lower display (and thus segment) to support touch, but not the main screen (other segment) given the gamer focus (cost saving) - I understand that on systems today these are considered different monitors, but AFAIU most of that was done due to missing platform APIs and this could change in the future.

Comment by @kenchris Dec 7, 2021 (See Github)

Can we clarify in the explainer that this doesn't cover regular multi monitor use-cases and only joined displays that are considered one virtual display/screen?

Comment by @kenchris Dec 7, 2021 (See Github)

In our TAG discussions we always end up with confusion around virtual displays (what the OS considers a display) that might actually be multiple physical joined displays. Can we please clarify that in the explainer and spec by linking to the relevant parts in https://webscreens.github.io/form-factors/

Comment by @darktears Dec 7, 2021 (See Github)

@kenchris In regards to your comment on foldables. With the refactor to support N hinges/folds it's getting harder to support that because there is no more information about the hinge/fold area. The segments property as described here will return two segments on a dual screen device where the geometries of each will allow you to layout around the hinge. Just like the https://drafts.csswg.org/css-env-1/#viewport-segments. I agree with your comment that if a foldable/OS decide to divide the viewport in two when the device is half folded, we could return three segments (two for the top/bottom and one for the fold area). Then it's up to the developer to decide whether to use a split UX or not. But in this case the "fold" segment must be defined to let the developer know where it is because of usability reasons (for e.g. avoiding putting a button or a scrubbing bar in the fold area) regardless of split UX.

I asked Microsoft in the past to add a property that would let a developer identify if the hinge is solid or transparent justly to cover that. Thinking about this proposal we'll have to expose the segments into an object that contains the DOMRect and the type (standard segment vs fold(opaque)/hinge(solid).

The fold case is something we should solve here and in the CSS counterpart. Our partners were specific around some of the issues with foldables: half cut videos, buttons in the middle of the fold are challenges that you see often. We need developers have the tools to address the issue.

Comment by @kenchris Dec 7, 2021 (See Github)

@darktears would some of this be helped if each segment would contain a bit of metadata, like whether it is the fold area or not?

Comment by @darktears Dec 15, 2021 (See Github)

I think it would.

Discussed Feb 1, 2022 (See Github)

Dan: I know there are multiple implementations. But all in Chromium. Different stakeholders are interested. Looking at S&P.. don't think theyre exposing any extra info. I think this is good, but I've been kind of working on it.

Peter: one concern is that the segments array is just a bunch of DOM rects. Comments between Ken and darktears that maybe it should have some embedded... should return objects, so room for expansion. Can leave comment. Other concern in general is that they seem to degrade very quickly into only dealing with one particular hardware situation, eg. screens are all the same size, or only two segments. This one doesn't suffer from that, the segments could be anywhere in space. No information aside from doing math with rectangles to figure out hwo they're oriented. Maybe there's metadata thtat can be added later.

Yves: related to second screen?

Dan: no

Peter: second screen is like casting. This is multiple monitors or foldable screens.

Dan: a few different hareware configurations this makes sense for. Gets very complicated if you start to hink about screens with multiple hinges.

Peter: I'm right now sitting in front of 3 monitors. There are existing groups of things that aren't just a foldable phone. I want it to not be treated as something different from a foldable phone. I think this one API is fine for that.

Dan: design is kind of born out of that concern. Intended to address that concern, to make sure whatever we do here is forward looking. Look again at plenary.

Discussed Feb 1, 2022 (See Github)

Rossen: both (689 & 690) are being experimented with

Dan: Peter asked a question about object vs DOM rect. No response yet.

Rossen: what would be the lifetime of such object? The beauty of rects is that they're immutable valid for the time being. If I return object now we're talking about bringing in the whole kitchen around the sink with events..

Peter: doens't need an event, doesn't need to be live, could be a snapshot. This isn't a blocker. Just a question. Negotiating API shape, overall happy with the feature.

Dan: Close with that question? Ping requester again?

Peter: worth completing the conversation

Dan: could close on wednesday if we get that feedback

Comment by @plinss Feb 21, 2022 (See Github)

Given a desire to add metadata, and for general extensibility, have you considered returning a 'segment' object rather than a simple DOMRect?

Discussed Apr 1, 2022 (See Github)

Peter: pending feedback for something minor

Dan: bump it?

Discussed May 1, 2022 (See Github)

Dan: we left feedback on 21st Feb and haven't progressed. As far as I understand it isn't very controversial, it is being used.

Peter: last feedback I left was about API shape. No response.

Dan: should we ping some of the other people on here? does so

Rossen: I pinged them Feb 28th

Comment by @torgo May 23, 2022 (See Github)

Hi @dlibby, @ststimac, @scottlow... we're trying to progress this review so we can get you some useful feedback and close it. Do you have any response to Peter's question from earlier in the year? Can anyone give us an update on the evolution of this proposal? Also where is this going after WICG? Any news there?

Comment by @diekus May 24, 2022 (See Github)

Hola @torgo, 2 out of 3 members there are not in the team anymore. Let me follow up internally and get you some much needed peace. Will report back soon.

Comment by @dlibby- May 26, 2022 (See Github)

Adding an extensibility point via a more complex object sounds like a reasonable way to stage this API. But I'm not sure I'm convinced unconditionally exposing the hinge as a segment is a net positive for developers, and either muddies the waters w.r.t. the concept of viewport segments as exposed via environment variables or makes them much more complex to use. Designing for future form factors is one thing, but consider the real-world devices in market today - Samsung foldables have no hinge, while Surface Duo does (i.e. the hinge consumes part of the viewport). Do we really want developers to have to add if statements and duplicate logic to support separate dual-view implementations on those devices?

From https://github.com/w3c/csswg-drafts/issues/6339, the VisualViewport spec is to be adopted into cssom-view, so CSSWG is the destination.

Discussed Jun 1, 2022 (See Github)

Dan: happy to "pass" this.

Rossen: I am too.

Thanks folks - on the basis of this feedback we're happy to close this and we're happy to see this work move forward.  Looking forward to a foldable future.

closed

Comment by @torgo Jun 6, 2022 (See Github)

Thanks folks - on the basis of this feedback we're happy to close this and we're happy to see this work move forward. Looking forward to a foldable future.