#575: Device Posture API [formerly Screen Fold API]

Visit on Github.

Opened Nov 23, 2020

HIQaH! QaH! TAG!

I'm requesting a TAG review of the Screen Fold API.

New types of mobile devices (phone and laptop segments) are coming to market with the ability to fold the screen in some capacity, either by using foldable displays or multiple internal displays (MIDs). Examples include the Samsung Galaxy Fold, Motorola Razr, Lenovo ThinkPad X1 Fold. 

While the technology enabling these devices is rather new, there is a resurfacing of a trend started by ultrabooks and 2 in 1’s which have had different operating “modes” or postures depending on the hinge angle. To note a few, “book”, “tent” and “tablet” modes were available on some devices. All these postures are triggered depending on the hinge/fold angle. The above specification would expose the fold angle as well as the postures.

From enhancing the usability of a website by avoiding the area of a fold, to enabling innovative use cases for the web, knowing the fold angle can help developers tailor their content to different devices. It can also enable to detect different postures the device might be in.

Further details:

  • I have reviewed the TAG's API Design Principles
  • The group where standardization of this work is intended to be done ("unknown" if not known): Devices and Sensors Working Group -> https://www.w3.org/das/
  • Existing major pieces of multi-stakeholder review or discussion of this design: Unknown
  • Major unresolved issues with or opposition to this design: None at the moment
  • This work is being funded by: Intel and Samsung

You should also know that...

Please also refer to https://webscreens.github.io/form-factors/ explainer for terminology and explanation of the problem space.

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

Discussions

2021-01-Kronos

Minutes

Dan: it's shipping behind a flag [on Samsung Internet].

Rossen: these scenarios are supported in Surface devices

Dan: privacy issue came up about knowing the angle. Some mitigations suggested and ongoing discussion on this issue. Talking about removing the angle itself and having access to different postures, rather than knowing specifically the angle that the screen is folded at

Rossen: that's closer to the previous MS proposal which had a few different modes that you're in, then based on those its' split screen or not, etc.

Dan: this work is ongoing, it may be premature to be doing a TAG review

Peter: I don't think so. I have some major issues with this, I think it's going the wrong direction. This has come up in CSS before. One is that this and almost everything else on foldable screens is very tightly coupled to existing screen tech and mobile devies. It's not forward looking at all. It's a matter of time before someoe makes a device with two hinges. An ipad with two smaller screens that fold out and aren't the same size, it's going to happen.

Rossen: more than one hinge as a forward looking issue, and that screen size is not necessarily homogenous in this example?

Peter: Yes. Third at some point in the future screens ize will not be static, rollup displays with pulling out as much screen as you want. It's been done in scifi so you know it's coming in reality.

Dan: there was an article about rollable screens

Peter: rolling or stretching

Dan: you could say that about almost anything that comes through w3c. I remember having the same argument about PWAs and the idea of a manifest and having an icon and peopel arguing that's so tied to the way people use smart phones, yes, but also there are many things about the way we use the web that are tied ot particular tech, liketouch screens, why have a touch api when we can be talking about generic interactions that could also allow for thought retrieval

Peter: that's pie in the sky, this is something we already have counter examples that are coming within a year. These APIs are very narrow and should be a bit more broad. My other concern is that it's not dealing with the desktop scenario at all. I'm sitting in front of 3 screens, two at angles. Why can't that be represented, why isnt' it in that API?

Dan: valid concerns. My concern, one of the reasons Samsung is interested, is how do we enable the developer access to that informations o they can make a choice ot create a user experience that fits that form factor. One of the things you can often do, that many applications that ship with the folding Samsung phones do is change the UI when you start folding the screen. There's an animated transition. When the phone is out and you're looking at the camear you might see the whole screen is the camear but when you start to fold it changes so the bottom is the controls and the top is the image. This is about enabling that on the Web. That's the goal. If we can comment back to the spec authors in a way that is encouraging and says if you take this into account you can achieve those goals while also being more forward looking to future display, that's very valid.

Peter: not just forward looking, past lookinga s well. I have a friend with 4 screens in a 2x2 grid. If the window spans the gap in the screens why shouldn't the web app know that?

Dan: this is different because there's a dynamic element to it. In the case I described you're in the same application. Your application might want to detec tthat and change the UI accordingly.

Peter: in a desktop scenario and drag the wndow the app would want to adapt in the exact same way, and use the same API to do that. Desktop is still a dynamic environment, it's the window moving not the display moving.

Dan: I see them as two different htings partially because of the mode of operation. Having it in your hand is a different kind of thing. This is a mass market case.

Peter: I don't want to pander to a short term mass market device with a lifetime of 2 years and ignore 20 years of legacy on desktop and 20 years of future mobile. The relatinoship between the app window and the screen is the same problem is whether it's the window moving on a fixed device or the device moving on a fixed window. It should not be a different api or developer experience or user experience for that.

Rossen: On principle I agree with Peter, in terms of future proofing, not as much as designing for the future as much as not blocking future designs by default. I'm not in favour of asking the authors to go and say tell me how it's going to work on a scroll. A little too far. But it should be fair to say if we had a scroll will this scenario prevent anything. Perhaps not. They can say if your scroll is not folding then this API is not going to kick in, we're not doing anything to prevent it.

Peter: in that case the screen seize is changing dynamically. I'd like to see an API for the screen topology not just the screen fold

Rossen: we're circling the same point, there is a lot of legacy and present that is developing, however you look at it a lot of the capabilities on what we have on the web today which are device centric, are driven by devices, we are always playing catchup with the current underlying OS. So we've been shipping OS capabilities for 2 years now that allow develoeprs to build native apps that take advantage of screen folds. So my question is how long do we keep the web from being able to use that? To hold our high bar of making the Web more pure. There must be a balance there where we allow the web to quickly catch to device capabilities while not fully handicapping ourselves for what's next.

Peter: I agree. My concern with the existing API shape is it is making presumptions about a single fold and a single direction and that's clearly inadquate. The other thing is this is device driven and thsee devices evolve rapidly and become obsolete rapidly. This is not goign to get to rec in 2 years and what's going to be new and shiny in 2 years?

Dan: I don't knwo if I agree with that. We've had stagnation in device design, industrial design, for years now. What I see happening is Apple is supposedly looking at foldable displays, foldable display itself is a new.. it's hard to say if it's a fad or not. Really hard to say. It is qualitiatively new and different. it has been proved to be manufacturable at scale. A scroll or wrinklable screen not so much.

Peter: we don't have to plan for arbitrairly wrinklable surfaces, but this doesn't plan for thing sthat have been proven to exists

Dan: multiple folds is valid feedback. We should be feeding back somethign that helps them but doesnt' block them because otherwise we are holding back out of purity

Peter: I'm not saying don't do this, I'm saying this approach is inadequate, so rethink it a little bit so we can move forward in a way that can be extended in a way that handles future tech and past tech that hasn't been considered yet like multiple screens on desktop. When some of these early fold proposals came to CSS it was around MS devics with a physical hinge, and that is a tech constraint that is likely to go away really quickly, so do we want to bake taht into an API for the web.

Dan: Leave some of that feedback on the issue and get the discussion going

2021-02-22

Minutes

Peter: Latest comment sugetst there's a second screen F2F taking place in the next few days. We should try to provide feedback before that. Remember a proposal by Microsft that allowed deailing with things like hinges, gaps, etc.

Rossen: It's not but this proposal does take the Microsoft Duo devices as part of its use cases. I also recall an issue about explicit angles.

Peter: My feedback is as before - this is too specific of a hardware use case and the API follows that requirement instead of a more generic approach.

Rossen: Have you provided the feedback? Seems like a valuable conversation for their f2f.

Peter: I will.

Rossen: Lea, are your happy with the responses to your questions?

Lea: Yes. The proposal is designed around current devices with one fold etc. The response was that the complexity of the fold etc. will become preventable quickly and they don't anticipate more than 2 folds. ... Had some more syntactic feedback that was addressed.

Rossen: How does this feature intersect other features such as orientation?

2021-05-24

Minutes

Still waiting for feedback, punte

2021-05-Arakeen

Minutes

Deferred to future

2021-06-28

Minutes

Peter: Dan asked if anyone else had anything on this

Ken: some is implemented in chrome by samsung, something something windows implementation. Has been interest from google..

Peter: last comment from alexis is that they significantly reshaped the api, do we have anything to add to that?

Rossen: trying to piece together..

Ken: used to be more like fold angle, used to be a way when the fold was changing to do animations, that's removed/postponed due to privacy. Other devices came up that didnt' necessarily depend on an angle so posture had to be made more generic, now called device posture not fold angle. Now it's basically exposed the name for the posture, and a way to query it. It's a big change

Peter: is that documented?

Rossen: explainer is now different... min and max angle, spanning. Is this spanning something harmonising with the spanning apis in css?

Ken: definitely harmonise but I don't think spanning is supposed to be part of this... using spanning from CSS yeah..

Rossen: High level feedback to explainer is that I'd like to see actual use cases here and examples instead of declaring the media. How people would use it would be interesting. Question about postures - one of the statements made by alexis is that the new proposed api doesn't have any type of problems if there's more than one fold. I wanted to know more about that. What does that mean?

Ken: that is becuase it doesn't talk about actual folds any longer it talks about postures

Rossen: if I have two folds and I fold the first one at 90deg what do I get at min angle?

Ken: it's not the angle any more. Exposing a posture which is a name.. that would just be flat posture

Rossen: Not what I'm seeing in the explainer

Ken: someone redid the explainer, it might be out of date

Rossen: go down to proposal: new css media query min angle and max angle... and an example. Then screen media posture, screen fold posture

Ken: is this sthe right explainer?

Rossen: maybe i'm looking at something else

Ken: explainer didn't get updated, I'll fix that. Also checking privacy review... I will tell alexis to fix this.

Rossen: okay so basically only the posture but not a whole bunch you can do outside of that? how do you target the rest of the functionality? with the spanning api or what?

Ken: the animation thing is maybe.. initial interest from samsung, when we tried to go over what they're doing in native apps no-one could find me examples. Everything else is with spanning

Rossen: so the span apis will give you the ability to know where the fold is, layouts and all of this stuff, but you still don't have the angle

Ken: you do't have the angle

Rossen: does that matter?

Ken: use case from angle is if you have a childrens popup book, things turn around when you open it, seems to be minor use cases.. and if it has to be exposed we have to make sure it works across if you have multiple folds and where. May have to be a javascript api. Then you get into privacy issues if you can query the angle all the time. With all of that together we decided not to pursue that at the moment. Unless people come up with real useful use cases for it. Also discussions around if you have multiple displays, how the displays are positioned toward each other.. some changes have happened with the multiple display spec.. the window segments is being integrated with that. We decided with this to do the bare minimum until everything falls into place, then maybe that will have to be combined with something like that

Rossen: I'd love to see examples that are real examples

Ken: someone did this new explainer and it's very short

Rossen: other point is the first motivation example on the very top has two folds, and none of the device postures is talking about this so it's very confusing for someone looking at the first time. Either change that, or..

Peter: how do you describe something that has two folds? add more postures?

Ken: yeah, additional or more descriptive postures, if it makes sense ot design for those

Rossen: this is a really great start, a great example of describing your feature and capabilities at the highest possible level and not exposing any additional lower level apis like angles or where things are, how they came about, velocity of closing/opening. If you really look at all of the possibilities for this set of functionality there are tons of them. on the native api we have in windows at least a ton of capabilities that are available that are a lot lower level extensibility of these models. Starting with this and going after feedback and more studies and more signals from both devices and end users of the api is the best path forward. I'm good with where this is going and how it's starting. Adding additional angles or whatever comes about later on is fine. I don't have any specific feedback on the postures themselves, they seem pretty straightforward although the only thing I'm questioning is how is flat different than no fold?

Ken: no fold here is supposed to be if you have a device that doesn't have any fold. Maybe that would be flat. But sometimes you want to differentiate.

Rossen: you can always add stuff. Removing stuff is harder. no fold vs flat I'm not sure which is better. Prefer single word values, so flat is better

Ken: questioning curved display... is that then flat?

Rossen: Crescent or watermelon...

Lea: mobius strip

Ken: It'd be great if you can give the comments on explainer, and file an issue with no-fold vs flat

Rossen: will add in issue

Peter: is this giving enough information to be useful? how would i actually change my webpage?

Rossen: if you go from a flat to a tent what you do know is that you now from a user pov have the intent to have only half of the screen visible.

Ken: useful for games

Rossen: you can split your webpage in two and flip them so one is facing one way and one the other way. This is where design should come and lead in terms of use cases. I wouldn't lead from the api to inform design decisions. Should be the other way around.

Ken: alexis did a battleship game like that with the tent mode. If it's flat you can see the opponant. Definitely games where tent mode makes sense

Peter: I'd like to see this driven by what uses it's enabling, how, and is it giving enoguh information to do that. If tent presumes split horizontally down the middle every time that may not be reality on all devices. I'd like to see actual use cases and how this enables those uses cases. I don't see it. Lacking context.

Rossen: yeah. Different types of fold and what use case they're motivating and codifying those in additional examples that are real examples is useful. I prefer this now because it is high enough level that allows quite a bit of extensibility late.r We're not pinning ourselves down to specific device capabilities. What if i have devices that can go into these folds but cannot provide angles? The lower level extension model here needs to be approached carefully.

Peter: agree. Not sure that this is solving a problem at this point. I'd like to see examples.

Ken: I'll get pepole to add examples. Not like every device will have to support all posture.s YOu might have one with a very small fold at the end, doesn't mean it's tent mode, that is up to manufacturer.

Peter: different ways things fold, I get that.

2021-09-Gethen

Minutes

Dan: One implementation (behind flag) in Samsung Internet right now.

Peter: ...in the example they showed a relatively modern device - that use case not covered. Existence proof of this device...

Rossen: one feedback - even motivational icon shows mutliple folds but current proposal iis for one fold. On flip side I've asked for more use cases... They pointed us to the spec which does have one but only for the single fold scenario.

Peter: I think the problem space is complex - and the feature is too simple. Not sure it's moving in the right direction.

Rossen: but isn't it? by popping their capability higher...

Dan: moving towards more semantic description for designers...

Peter: but what do I do with that? How do I know where the fold is?

Rossen: this is addressable. The base : we have capabilities that are exposing pieces of screens - hinges, seams, magnets.. etc... this current one is not talking about this. It's describing the postures of devices with screens one or many...

Peter: there are other specs that will provide that info?

Rossen: yes - the media query

Peter: i know there are multiscreen apis - how does this fit with the other specs?

Rossen: this (high level) with the spanning capabilitiy - where essentially you can

Peter: if I have to go to 2 or 3 other APIs can't I get the info this is giving me fromn those APIs as well? Is this useful?

Rossen: you could ask the same about @media print. You know that you're in that environment.

Sangwhan: what media queries?

Rossen: a few different environemnt variables describe where the fold is...

Sangwhan the fold is that you have a foldable screen.

Rossen: it's the same thing - a tent - you know where your origin is and know what your width and height is...

Rossen: is there a media query or function today that describes this?

Dan: so would it be good feedback to ask they to explain in their explainer or spec to be specific about how authors are supposed to use ths API together with other environment variables and othert mediat queries?

Rossen: they've answererd that question. See https://w3c.github.io/device-posture/#example-2-adapting-ui-to-posture

Peter: Can't judge the usefulness of this - is it giving me useful info.

Sangwhan: Agree.

Rossen: Kind of neutral - what this bubbles up to - you have the high level media trigger - this device posture changed - there may be posture chanegs where you don't care. From the pov - this top level switch is useful. In the examples they give in the spec the device posture alone is not useful because you also have to look at the spanning - so you could describe everything with the spanning. When I worked on this with the folks that produced the spanning - nobody used e.g. portrait and landscape because people used the metadata. You cannot react to the posture alone and adjust your layout.

Peter: this adds one and a half bits of info - does that exist anywhere else...

Rossen: i don't think the answer is yes.

Sangwhan: I have the same question...

Peter: in their example they say "the spanning and the posture" - do you need the extra bit of the posture.

Rossen: if you go from a laptop - and fold it flat - so one big screen - spanning works so you change the origin and orientation of both screens.

Peter: this adds tend mode - folded over... Then I'm looking at both screens. If the spanning did not change then it adds useful info.

Sangwhan: There is a big picture here but we are only being shown a piece of the puzzle. I think we should ask for a review at the point they have the landscape sort of planned out. Either that or a landscape explainer.

Rossen: changing from screen to page you have to restyle... here you're notr eally changing.. By abstractig the spec to this level - lost the angle.

Peter: i think it's moving in the right direction...

Dan: I think asking for a landscape document.. sounds like a good idea.