#473: Delegated Ink Trail

Visit on Github.

Opened Feb 14, 2020

Hello TAG!

I'm requesting a TAG review of Delegated Ink Trail

Achieving low latency is critical for delivering great inking experiences on the Web. Today, ink is generally produced by consuming PointerEvents and rendering strokes to the application view, but this introduces a good bit of latency by itself. The goal of this enhancement is to reduce this latency as much as possible, by providing a compact API that can add a few extra points to the end of the ink stroke that has already been rendered. This will be achieved by forwarding input directly to the display compositor. Then just before the back buffer is presented any points that haven’t already been rendered by the application can be applied and connected to the stroke already rendered. This, combined with potentially predicting a point or two ahead, can reduce perceived latency and provide a progressive enhancement over other latency improvements.

Further details:

  • [✔] I have reviewed the TAG's API Design Principles
  • The group where the work on this design is being done (or is intended to be done in the future): PointerEvents WG
  • Existing major pieces of multi-stakeholder review or discussion of this design: https://github.com/w3c/pointerevents/issues/204
  • Major unresolved issues with or opposition to this design: None
  • This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify @mabian-ms

Discussions

Comment by @torgo Feb 26, 2020 (See Github)

Discussed briefly on our call today. We noted it might be useful to have a more basic user needs description.

Comment by @dbaron Feb 26, 2020 (See Github)

@alice noted the relationship to #346 and the getCoalescedEvents proposed there.

Comment by @dbaron Mar 4, 2020 (See Github)

@atanassov and I are looking at this briefly in our Wellington face-to-face.

One thing I'm wondering is why the requestPresenter function takes a type as a string, rather than just having a method that returns either Promise<DelegatedInkTrailPresenter> or just DelegatedInkTrailPresenter. It's not clear to me what the reason for this string-based extensibility mechanism is or how it's intended to be extended in the future. And what is the abstract InkPresenter intended to extend to in the future? In general, extensibility mechanisms seem sensible if there's a good idea of how to extend it in the future; it would be good to get an understanding of what you see that extension as being so we can help evaluate if this API makes sense for it.

It would also be interesting to understand why Promise is used here; what potentially requires it to be asynchronous?

I'm also wondering why the radius is an unsigned long -- that seems worrying in that more fractional accuracy might be desirable. It seems like it should either be consistent with canvas which tends to use unrestricted double (probably more likely) or consistent with CSS and use length strings. It's also not clear to me what the radius is measuring.

It's also not clear why the Element argument to requestPresenter is optional, and what happens if it's omitted. (And is that element argument always the presentationArea of the result? Seems likely, but perhaps worth saying.)

I'm also a little worried about the naming of these functions, in that there isn't really a function call whose name suggests "go ahead and do ink drawing for everything that's happened since setLastRenderedPoint". A call to requestPresenter perhaps doesn't sound like it's going to do that. (And I think that's what makes it happen, although I might be misreading.)

I'm also a little concerned about the usePrediction = true example. Can't the prediction be wrong? Doesn't it then require that the example do more work to clean up the incorrect prediction data? Or do I misunderstand what "prediction" means here.

Comment by @dlibby- Mar 6, 2020 (See Github)

Thanks for the great feedback, much appreciated!

The string parameter to requestPresenter is indeed intended to be extensible in the future, but as you pointed out, we didn't explain any of that (whoops). I filed an issue in the WICG repo (where the explainer has moved since the TAG review request was made) - we do think there could be room for more InkPresenters in the future. The delegated ink trail in the proposal is the simplest form of this, but you could imagine authors wanting similar functionality but with the ability to render the trail with shaders. That being said, it's possible that we're overthinking the extensibility and can start with something more straightfoward.

The request returning a Promise is a side-effect of the design for extensibility. It's possible, e.g., that something like the complex trail would only be supported if hardware rendering is enabled, or some other criteria that is not immediately available.

Good point on the radius type, and also the lack of clarity. I filed an issue to address this in the explainer. The radius describes the stroke weight at the last application-rendered point so that the UA or OS can match. Inking applications can use pressure to change this value as the user moves the pen in different ways.

I agree the naming could be improved - as it stands there's somewhat of an unspoken/implicit agreement that setLastRenderedPoint will result in the ink trail being rendered outside of the application. Filed an issue to come up with a better name.

The optionality of presentationArea is explained in a paragraph that is not clearly marked - the area would default to the viewport. Yes the presentationArea attribute would be the argument passed to requestPresenter. Filed an issue to clean this up.

I can add some more details regarding the prediction bit. This comes in to play in the fallback scenarios, where the application doesn't want to (or can't) use this API. Typically applications will either implement their own prediction, or use the fairly recent PointerEvent.getPredictedEvents() function to render ink segments past the last received pointer event. The usage here was mainly to highlight that in these fallback cases, the author has the option to use prediction to decrease perceived latency, where if they do use the API, the application should not try prediction, as the user will see the (potentially incorrect) predicted rendering right next to the ink trail, which will be correct since it uses the real input from a later point in time.

I'll work on addressing these points in the explainer over the next week or so and ping this issue when I'm done.

Discussed Mar 23, 2020 (See Github)

Rossen: I moved the issue out by couple of weeks (to April 6) as the feature authors are actively addressing the feedback provided by David and myself during last round of reviews in Wellington

Discussed Apr 6, 2020 (See Github)

Rossen: David and I looked at this in Wellington. We left detailed feedback that was largely agreed upon with the authors. I pinged the author and he said the current version of the explainer is ready for review, and it has incorporated the feedback.

... If you go to the current explainer that's proposed, you'll see this one is marked as archived, because the explainer moved to a WICG repo

... we've adopted this document status policy for all our (msft's) explainers

David: I've updated the issue to use the latest URL

Rossen: Are you able to follow up with the feedback you left, to see if the current version addresses it?

David: It looks like he filed a bunch of issues from our feedback and those issues are resolved; i need to look at the issue resolutions to see

Rossen: he did a good job of linking to the PRs that addressed each one.

... [gives example]

Tess: push this out a week or two?

David: Sure

[pushed out one week]

[rossen drops

Discussed Apr 13, 2020 (See Github)

David: comment from dlibby - said will work on addressing points.

Rossen: all of the edits are in - all addressed in WICG in terms of PRs. As far as the authors understand it, the ball is in our court.

David: Ok - i will do this and let's bump till next week.

bumpe

Comment by @dbaron Apr 13, 2020 (See Github)

It sounds like all the issues have been addressed an this point, so we should have a look at those resolutions; will try to actually make that happen by next week.

Comment by @dbaron Apr 17, 2020 (See Github)

So a few responses to those issues:

  • diameter still doesn't say what sort of units it has. Is it CSS pixels? Device pixels? Something else? And should it really just be a number, or a string that accepts all sorts of CSS lengths with different units?
  • for presentationArea, if it's an element or null, then it should probably be written with a question mark as:
    readonly attribute Element? presentationArea; 
    
    Likewise, the argument to requestPresenter should probably be optional Element? presentationArea = null. Also, if "bounding box" means border box, perhaps it should use that term? If not, what does it mean?
  • I'm not particularly convinced about the extensibility, but I'm also not sufficiently unconvinced that I'd strongly argue that it shouldn't be there.
Discussed Apr 20, 2020 (See Github)

Rossen: David still has some open questions in the issue

David: but he did miss the points about diameter and what units it is in. Also Web IDL errors.

Ken not a big fan of them doing prematurely extensibility - added comment to the issue.

David: Fine with closing, no strong opinions.

Rossen: Same - but too close to the issue to make the call.

Ken: Let's defer to the plenary

Comment by @kenchris Apr 20, 2020 (See Github)

I don't see the point of the premature extensability. You can always make it inherit from InkPresenter in the future if such a need would arise. I don't think that is going to be a big breaking change as people shouldn't rely on it not inheriting from it or it not inheriting from anything

interface InkPresenter {
}

interface DelegatedInkTrailPresenter : InkPresenter {
   ...
Comment by @mabian-ms Apr 21, 2020 (See Github)

Thanks for the comments! I've addressed them in the explainer, but to clarify here:

  • We've been prototyping with diameter being in CSS pixels as that is what felt most intuitive, so I've made that more explicit. I don't think a string would be necessary for diameter, as I think the number of cases that it would be easier to use than CSS pixels would be very limited in scope, and not necessarily much harder to do in CSS pixels.

  • Good catch on presentationArea IDL syntax, fixed that. Also clarified that it is indeed the border-box in client coordinates that we use.

  • Extending the API to allow for shaders and other more complex brushes came up organically on the Chromium graphics-dev post discussing this API, so I don't think it is too difficult to imagine wanting to extend this to support these in the future. Given this, and that there isn't a compelling argument that it shouldn't be there, I'll leave it for now. Happy to continue discussing it if necessary though.

  • I see what you mean about the premature extensibility - that makes sense as it can be painlessly extended in the future. I've removed the InkPresenter interface.

Comment by @dbaron Apr 22, 2020 (See Github)

In any case, while there's still a bunch of detail to be worked out, from the perspective of an early design review we're satisfied with this.

We'd encourage you to continue working through the feedback that you're getting from us and other sources, and to get wider review from developers and from other implementers as a path towards potentially getting it into a standards track.

Comment by @mabian-ms Jul 8, 2021 (See Github)

Resurrecting a bit of an old issue at this point, so please let me know if this should be in a new issue. Thought I would start here for the context though.

I'm working on a draft version of the spec to launch this feature in Chromium soon, and one topic that has come up again is the string type param on requestPresenter. Given that it is only there for future extensibility, it is essentially a no-op parameter right now. Therefore, there has been some talk of potentially dropping both parameters in its current form and using a dictionary as the only param instead, which would only optionally contain a presentationArea. That way we could freely expand it to include a type in the future.

An updated IDL for this would then be something along the lines of:

dictionary PresenterParam {
   Element presentationArea;
 }

 Promise<InkPresenter> requestPresenter(optional PresenterParam param = {});

The concern that I have is that it seems like it could be a confusing or unexpected experience for developers in the future when the type param is added. Then they could get the delegated ink experience via either specifying type: delegated-ink-trail or leaving type out completely. This isn't a blocker, of course, but two different ways of achieving the same result seems like something we'd prefer to avoid.

Does TAG have any preferred pattern to follow in situations like this, or a stronger opinion on the type param?

Here's the link to the issue on the draft spec, for a bit more discussion: https://github.com/WICG/ink-enhancement/issues/20

Comment by @domenic Jul 8, 2021 (See Github)

IMO (not TAG) having default values for dictionary members, so that when you leave them out they behave as some specific value, is totally normal on the platform.