#528: WebXR Layers

Visit on Github.

Opened Jun 19, 2020

Saluton TAG!

I'm requesting a TAG review of the WebXR Layers specification.

WebXR Layers offers a more efficient way of drawing immersive content than WebXR. In addition to support for native textures and texture arrays, it also provides support for different layer types that are managed by the system compositor (as opposed to javascript). This spec is an optional module that extends the feature set of WebXR.

Further details:

  • I have reviewed the TAG's API Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): W3C Immersive Web CG
  • The group where standardization of this work is intended to be done: W3C Immersive Web WG
  • Major unresolved issues with or opposition to this design: none
  • This work is being funded by: Facebook

We'd prefer the TAG provide feedback as: 🐛 open issues in our GitHub repo for each point of feedback

Discussions

Discussed Jul 13, 2020 (See Github)

[bumped to B

Comment by @torgo Sep 22, 2020 (See Github)

Hi @cabanier - We are just in the process of reviewing in our virtual f2f. One quick suggestion: I feel like the explainer would be a bit more explainy if you took the firs paragraph from the "use cases" section and moved it to the very top of the document – that way it could provide important context for what I'm reading right under it.

Discussed Nov 9, 2020 (See Github)

Present: Dan & Rossen

Rossen: From previous meeting, we do like the general direction and this session is about digging further into the spec and all functions that it brings with it.

Dan: on security & privacy questionnaie, they have not answered it, pointing only to answers being "in the spec" but there is a small one-item securty & privacy considerations section. Will ask them to expand. left comment

Rossen: When going through explainer it was difficult to understand what other options were considered and why these ones are the best. Perhaps they can add a section with that explanation.

Dan: loving the diagrams in the doc

[discussion of the nativeProjectionScaleFactor]

Rossen: this isn't something we've had elsewhere in the platform... I don't believe we have a scale factor today that would give users the ability today to let users understand what the scale factor is for a layer or view. Could be effected by either the user (e.g. pinch zoom) or the content. As an author I don't have access to this info from the platform. This API will enable that but in an awkward way - creating a webGL binding etc.. Meta point is we're either missing a way to bubble this up a bit further as a more generic platform capability - more than just the webxr context - or be very careful here and maybe not expose it. These have been a pain in the platform - e.g adjusting content to prevent scroll bars - capability has been sought after since forever.

Dan: also - a potential fingerprinting issue.

Rossen: proogogating anything up from the underlying platforms should be done carefully. It's not obvious to me if there are platform layer violations in this spec. Another main scrutiy & privacy concern: how easily and quickly would such capability allow readbacks from unintended actors? Just because my bank added a twitter logo and pulled in some twitter iframe - i don't want them to be able to get a hold of my layer buffer.

Discussed Nov 9, 2020 (See Github)

Rossen: we said "this is great" on face value - we need to go and look at the details. I will leave a comment. We need to spend a good time reviewing this.

[bumped to next week, rossen & dan to have breakout]

Comment by @torgo Nov 13, 2020 (See Github)

Hi @cabanier. @atanassov and I are just reviewing now in a special break-out. I noticed that you didn't answer the security & privacy questionnaire, opting instead to point us to the spec, but in the spec there is only a very short privacy & consideration section. Can you please try to answer the actual security & privacy considerations questions (in a separate doc - doesn't need to be fancy - a text or markdown doc is fine)? I think there may be more factors you need to consider then what you have listed in the considerations section.

Comment by @cabanier Nov 13, 2020 (See Github)

Thanks Dan! I will do so today.

Comment by @atanassov Nov 13, 2020 (See Github)

@cabanier one more editorial feedback about the explainer. The two introduction sections offer little motivation to users who aren't deeply familiar with WebXR, rendering or composition. It will go a long way if you can add a bit more details to that end.

Towards the end of the explainer there is usually a section that describes other approaches that were considered and why this is one the best. Given there are so many capabilities in this spec I can see why that could be difficult and hope you can offer at least some view into your decision making.

Comment by @atanassov Nov 13, 2020 (See Github)

@cabanier couple of more questions as we read the spec.

It seems that exposing nativeProjectionScaleFactor will enable authors using the feature to obtain the scale factor for a given context (assuming post transform?). If this is the case, my assumption is that such capability will be very sought after many other places in the platform (users of CSSOM have been asking for something like this forever). Could this be either omitted or made available outside the WebXR context?

Have you thought through some of the threat vectors that could abuse such capability? How quickly and easily could a malicious actor create a WebXR context in a different content and obtain the composition layers?

One final meta question. Have you considered, or at least do you think it is feasible to break down this review into few logical pieces that build on top of each other? Layer Types, Spaces, Creation, Rendering etc. That model of reviews will be much more expedient to all of us.

Comment by @cabanier Nov 13, 2020 (See Github)

It seems that exposing nativeProjectionScaleFactor will enable authors using the feature to obtain the scale factor for a given context (assuming post transform?). If this is the case, my assumption is that such capability will be very sought after many other places in the platform (users of CSSOM have been asking for something like this forever). Could this be either omitted or made available outside the WebXR context?

nativeProjectionScaleFactor is the value of scaleFactor that was passed in during the construction of the projection layer. It's basically the same as the framebufferscalefactor value in the WebXR spec It is NOT to the "scale factor for a given context (assuming post transform?)". WebXR Layers can only be used in immersive sessions so it has no relation with what is rendered on the page.

Have you thought through some of the threat vectors that could abuse such capability?

There's no new capability with the scale factor. The system will just allocate a buffer where the native buffer size is scaled by this number.

How quickly and easily could a malicious actor create a WebXR context in a different content and obtain the composition layers?

It wouldn't be possible. Immersive WebXR sessions can only be created after a user action and consenting to go immersive. This is not something new that the WebXR Layers spec introduces; this is already covered in the WebXR spec.

One final meta question. Have you considered, or at least do you think it is feasible to break down this review into few logical pieces that build on top of each other? Layer Types, Spaces, Creation, Rendering etc. That model of reviews will be much more expedient to all of us.

As I mentioned earlier, this spec heavily depends on the WebXR specification so in order to review it, one has to be familiar with that one as well. I am planning on adding examples and non-normative explainers to clarify the intent. Right now it's mostly a list of interfaces and flow diagrams which makes it hard to follow.

Comment by @cabanier Nov 13, 2020 (See Github)

Can you please try to answer the actual security & privacy considerations questions (in a separate doc - doesn't need to be fancy - a text or markdown doc is fine)?

I added a markdown document here. Please let me know if I need to elaborate more.

I think there may be more factors you need to consider then what you have listed in the considerations section.

Apart from timing of the composition, I'm unsure if there are any other considerations. The spec at its most basic level just exposes new textures to render content to. I'd be happy to have a conversation or to add more to the specification to clarify.

Comment by @cabanier Dec 5, 2020 (See Github)

@atanassov @torgo Do I need to do anything else to have this picked up again by the TAG?

Discussed Dec 7, 2020 (See Github)

Rossen: My take-away is that everything we asked for has been thought of... They are building closely on top of WebXR... There is essentially nothing new that's been added that's been concerning from API exposure. on scale factor - .. he got back to us ... he's saying it's the value of scale factor in consutruction... not what we were thinking.

... a few questions around security & Privacy - he did fill out the questionnaire...

... we asked him to possibly split it. He cited a heavy dependency on webXR to push back. I agree for the spec, I'm not sure about the explainer...

... they may want to break it down to new capabilities...

... the spec itself is split into 6 different logical submodules... lifetime, initialization, type system, event model, rendering model...

... it's marked as "early" but maybe not so early...

Dan: what are you suggesting?

Rossen: the explainer does a good job of reflecting concepts from the spec, but doesn't answer some key questions - goals and non-goals, other potential solutions considered, ... which parts of the examples are newly added capabilities vs new functionality...

... meta-question is do we "hold them back" ?

[taking a look at explainer, s&p questionnaire, etc...]

left comment

Comment by @torgo Dec 7, 2020 (See Github)

@cabanier we're just reviewing in today's TAG call. Thanks for filling out the security & privacy questionnaire - much appreciated. We have no red flags per se, but this is such a big piece of work that we're worried we're missing something. We could do with some more detail in the explainer - specifically "goals and non-goals, other potential solutions considered," also it's not clear to us which parts of the examples are newly added capabilities vs new functionality. Would it be possible for you to join one of our calls in the new year to go through it?

Comment by @cabanier Dec 7, 2020 (See Github)

Certainly! I will work on cleaning up the explainer.

Comment by @cabanier Dec 8, 2020 (See Github)

@torgo FYI I added "goals and non-goals, other potential solutions considered" to the spec and a couple more clarifications.

Comment by @torgo Dec 9, 2020 (See Github)

Thanks for this Rik - we are generally positive - we're going to spend a little bit more time on the security & privacy review and we will try to close this one off very early in the new year.

Discussed Jan 1, 2021 (See Github)

Rossen: The last comment added by Rik is what was missing. With that information we can have a clear expectation of the texture lifetime management and why this isn't a vector for security concerns.

Rossen: propose closing this.

Dan: A-OK

Discussed Jan 11, 2021 (See Github)

Rossen: ability to create layers - which allows for optimisations etc.. in applications such as gaming, whatever... tons of details here. Huge spec. Extension to XR. APIs being added for construction, lifestyle, etc... It's a great addition to the platform but devil in the details. One question was whether they can break into pieces. They prefer not to. We asked if it's an eraly review - from our PoV early - may already be shipping in some places - where we are today is we asked for privacy & security review. We felt the overall direction is good and we are ready to close. We were going to spend more time on security & privacy.

... they have a doc ... They are falling back on "this is an extension of webxr".. so not so much added. Not 100% agree on security because they are adding new object lifetime management - if t's tied to the GPU then I can speculate that the security part is more complex.

Dan: Yeah in their response they didn't go into too much detail.

Rossen: i did post a question about malicious actors - mostly around the privacy side. which was a long way to say - can I obtain visuals from the hosting document and capture what you're seeing? They said "this is not possible". I have confidence in the response.

Dan: it should be in the security & privacy considerations section.

Rossen: I will ask for a bit more detail on lifecycle management of objects and primitives in the compositor. And we can propose closing after that. We're happy with what they're doing.

Ken: they should spell out the mitigations.

Comment by @atanassov Jan 11, 2021 (See Github)

@cabanier, we had another look at the security & privacy document, thanks for completing. One question and concern that came up (security more than privacy) is around the lifetime management of layers. Can you elaborate on the coupling between the layer objects and underlying GPU textures or other lower level primitives? How is this proposal ensuring that lower primitives are protected from malicious use?

Ideally this can go directly into the the security & privacy document or the explainer.

Comment by @cabanier Jan 11, 2021 (See Github)

@atanassov can you elaborate your concern a bit? The textures are defined to only be valid during a Raf call and after a call to getSubImage/getViewSubImage. If you use them at any other time, they are considered invalid. Is this what you are looking for in the explainer?

Comment by @cabanier Jan 13, 2021 (See Github)

ping @atanassov

Comment by @atanassov Jan 27, 2021 (See Github)

@cabanier sorry about the delay. @torgo and myself had this scheduled for completion during our current "Kronos" VF2F.

The textures explanation above is what I was missing. If you don't mind adding it to the security section of the spec and/or the privacy and security document.

With that added we are happy to close the review as satisfied.

Comment by @cabanier Jan 27, 2021 (See Github)

Thanks @atanassov! I updated the explainer with that information and added some more details. Let me know if you still have questions or concerns.

Discussed Feb 15, 2021 (See Github)

Rossen: ... asking Rick to add his explanation / concerns to the actual spec or questionnaire...

Dan: I do see the change but not

[realized it's a branching issue]

Dan: [updates the links in the issue]

Rossen: OK I'll write the closing comment.

resolved to close

Comment by @atanassov Feb 15, 2021 (See Github)

Thank you @cabanier! Took me a bit to see the change in your explainer only to realize the link here was to 'maser' not 'main' branch :) Anyway, everything looks good and we are excited to see progress being made at W3C Immersive Web WG. Thank you for opening this review and working with us, good luck.