Luke: Looks like a replacement request.

Matthew: From last week, didn’t see it yet. Jeffrey put a comment that this replaces design-reviews#997.

Luke: Dubious of privacy impacts on painting on the canvas. Because it can read back pixel data, it might lead to fingerprinting. Think accent colors in form elements. That is being handled, but still, many things could be left over.

Yves: There were timing attacks in the past. Can be bad. Needs care.

Hadley: Luke, if you’re interested in this, read the design review and participate in the private brainstorming thread.

Xioacheng: This is a new version of a previous proposal. I am very interested, but at the same time I think it needs a lot of work. First concern: privacy. We raised privacy concerns previously. What kind of data an be drawn into a canvas. In the current explainer they listed a range of things they considered. They also listed things they can't consider regarding privacy, like scrollbars etc.

...I am concerned they're not doing enough. I'm wondering if they have higher-level criteria for judging what's sensitive and what is not. The list is incomplete - we may find later that things should've been considered privacy-sensitive.

...the paint event is a new event. The idea is whenever the rendering of a child of the canvas changes, a paint event is triggered. Then the listener can update the image in the canvas. I think we need more work on the kind of changes that can trigger this paint event. There are some changes that might not trigger the update of the rendering. E.g. video playback, or image animation, or just a blinking caret. These kinds of changes... not sure if they're completely exposed to JS. UA can do lots of optimizations to put them on other threads. I am wondering if there would be interop risks, e.g. if some new implementation offloads work to a differnet thread, but we require them to change it.

...Also concerned as to whether we want to keep the descendents as interactive, or how to achieve interativity, Current approach seems hacky. The draw element-image method returns a transform matrix, and then put that matrix on the element's styles so it's transformed. This feels both unergonomic, and error-prone. What if we draw the same element twice? What if there are multiple elemlents but we draw them in an order that's different to DOM order?

Lola: You said that using the trasnform matrix and applying to element is hacky. If we draw the same element twice, what is the effect? Wouldn't we want the same thing to happen to both of them?

Xiaocheng: There's no way to apply one transform to one of the elements only - e.g. when we are doing hit testing.

Lola: Makes sense.

Matthew: APA is looking at it.

Matthew: Still looking at this.

Luke: I think it's a problem worth solving; solution seems to have been thoroughly reviewd to ensure it's implementable. I think the main thing is nailing down whether the accessibility works as intended. And getting the privacy stuff nailed down so that all the engines are on board with what they consider worth redacting, and what's OK to leak. There was discussion on the WHATWG calls that some things are leaked becuase they are already leaked, but there was some disagreement that that's a reasonable approach to take.

Christian: To be continued. Feels pretty big. So we are not at the point to say 'yes' or 'no'

(With Philip Rogers calling in.)

Philip: Responding to the minutes. When you use canvas, you opt out of all the default web platform features, like accessibility, selection, zoom, translate, agents. HTML in Canvas enables authors to use the web platform in canvas. So they don't need to reinvent them. Comment around privacy implications. Unfortunately, there's not a high-level design to use here. Case-by-case. e.g. forced-colors mode. Could choose to expose it or not-expose it. Is the new information and fingerprinting worth the benefit to users. In none of the cases is it an easy decision. For forced-colors, it's already available in canvas, and we expose it. But others are case-by-case. How we got where we are. Started with the design engines use for drawing SVG. Can hack using SVG foreign objects. Browsers have had to deal with it. I did it in chrome and WebKit. Built up knowledge. Have to re-evaluate, but can start there. Looked at how things rendered in different engines. Thought about everything with this tradeoff: privacy, cors, revealing themes, etc. Made a decision for each one, and iterated. Internally, with Google security team, spec process, and with AI. Ended up with the list in the explainer and spec. Not perfect, but that's how we got there. What do you think?

Xiaocheng: That sounds very nice. You mentioned a high-level principle of the tradeoff between benefit to users and privacy risk. This is reasonable.

Philip: History wasn't available when you reviewed, and it's been changing. Current ability to draw with SVG foreign objects is widely used. Doesn't allow drawing interactive content. So interactive privacy issues are all new. Another small note: we filed a spec issue to take the findings from this, and use this for the foreign-object security model to improve interoperability for SVG foreign objects.

Xiaocheng: Think the foreign object hack was mentioned in the explainer. Very important, and we want to do better.

Philip: Some people wanted to take it away after we learned about it, but the use counters say it's too popular.

Marcos: Isn't that somethingw e should fix at the web platform level? Should be uses for canvas for certain things, and sometimes you want buttons, but if there are other cases where we should push up the chain, due to interop issues and rendering text. Shoudln't be in the domain of canvas.

Philip: If you draw foreign-object into canvas, there are browser differences around, e.g. scroll bars. We should all agree about thre tradeoffs for that case. Kind of thing we want to bring back around foreign-object. If we agree on html-in-canvas tradeoffs, we can tighten interop for foreign-object too.

Marcos: They'll look different regardless due to platform differences. Want to keep those. Having those cases pointed out ... e.g. draw whatever you want as a scroll bar but make sure it's at least this size. Interop: have to weigh against privacy implications. Fingerprint pretty heavily with this. Might be enough motivation to shut it down. Hopefully not.

Philip: Fingerprinting issue is top of mind. Pragmatic issue: it's easy to jam a bunch of stuff in canvas, and hash the pixel data. We shoudl be careful. Most of the issues here are already available. You have text drawing in canvas, so you already get fonts. Shaping. Exactly how glyphs look, font fallbacks. Not to say we should stop here. Those are out of the bag. Design principle for html-in-canvas is that we should be able to lock things down on a case-by-case basis. If, e.g. ligatures aren't ok, we can disable that for rendering html-in-canvas.

Xiaocheng: How bad is the privacy issue for foreign-object?

Philip: Not a lot of information from foreign-object that you can't get from drawText(). I'm not authoritative, but drawText() API is the primary fingerprinting issue. New information like buttons isn't bit.

Xiaocheng: visited colors?

Philip: We lock them down in foreign objects and HTML-in-canvas.

Jeffrey: The ability to lock things down one at a time makes this feature align with https://www.w3.org/TR/privacy-principles/#unavoidable-information-exposure.

Philip: Interactivity: If you expose HTML in canvas without interactivity, it's dangerous in a lot of ways. E.g. accessibility. If you draw a button that's live in DOM, accessibility tree thinks it's live, so it should be live in canvas. Text selection also needs to work end-to-end. Overall issue is top request from people prototyping this early. Drawing forms into canvas is extremely hard. Tracking typing, backspace, arrows, needs building a whole browser.

Xiaocheng: Understand the motivation. Is this really achievable, or do we need more restrictions to make it feasible? E.g. interactivity is tricky ..

Marcos: Poor PEPC people want to render browser-secure UI, and it wouldn't work with this. When you drop it, it reveals permission state. Can't render it. Assume they've come to you. On WebKit we can't use PEPC buttons in canvas since it circumvents whole PEPC thing.

Philip: We have an answer for that class. There are similar issues with, in a normal <select>, with the popup drawn by the OS. Popups don't draw into HTML-in-canvas. Text IME doesn't draw into canvas. Browser security popups. Nothing cross-origin.

Xiaocheng: Another canvas?

Philip: Another canvas is ok, but we lock down nested html-in-canvas.

Xiaocheng: Interactivity: apply transform matrix to element. Feels hacky. Wondering in general if interactivity is feasible. What if we draw it twice? Canvas has multiple child nodes, drawn in non-DOM order?

Philip: Today, if you draw an element twice, and we return a transform from the drawElement API, and there are equivalent ways to do it in WebGL and WebGPU. Then we update the DOM's transform to match. Only 1 DOM, so only the last is hit-testable. It's an anti-pattern.

Xiaocheng: In one of the demos, something is drawn ...

Philip: We have a cube demo, which we should remove. Antipattern. That breaks the whole idea of accessibility. Which is accessible/live? This API has grown, and this example is carried over from an old version. Another issue was the ordering issue. That's tricky since you need to sync the DOM order with what you've drawn. Have a proposal in the issue to make that easier. When you're drawing an element into canvas, you can make the decision about the transform, but can't decide about the DOM ordering. Doing this nicely requires a follow-up change. Have a pretty clear, but it's a problem now. Another issue was that transform sync'ing seems hacky. Frameworks: if we automatically set the transform, can break virtual DOM. Need author to be in the loop for the transform setting. Also trying to build the lowest-level primitive. Possible today with filters. Also can draw 0 times with opacity:0, which is hit-testable but not visible. Not good, but worse to try to lock down the API super hard. E.g. drawing element twice: if someone's trying to make a sprite-sheet, it would break the use case. Not locking things down until we see problems and what authors are doing. There is a footgun. Even if you lock it down fully, you can draw to canvas, read, and then write it again.

Xiaocheng: IIUC, goal here is to maintain interactivity under "correct" usage.

Philip: Interactivity by default. Another example: browser can translate. Without interactivity, if the user clicks 'translate', text won't translate. Need to have the whole thing hooked up to have a real web platform in the canvas.

Xiaocheng: Good answer. More hypothetical: we restrict interactivity to correct usage. Will this lead to, in the future, people say this isn't interactive enough, and let's add a new version? E.g. HTML in canvas is interactive only if it's used correctly. In the future, people find incorrect usage important and also want to support interativity. How likely?

Philip: Very possible. E.g. a game uses a sprite-sheet to stamp out buttons. Now they want to make the buttons live? Encourage author to update the DOM and have multiple buttons in the DOM. When authors can make DOM match, that's ideal.

Brian: Things you're saying aren't drawn to canvas, are htye not drawn period, or are they potentially layered above the canvas and not read-back-able? Things you've marked sensitive, like pseudoclass spelling/grammar error aren't read-back-able even in CSS. I can imagine still drawing them, because the use cases you're describing, editing+selecting text, if you don't have those, people will recreate the universe to get them.

Philip: Get that. If you can't draw spellcheck markers, that's clearly useful. Each of these has to be considered case-by-case. When they're drawn into canvas, they're fully exposed. Can't use tainting that's sometimes used for cross-origin things. Spell check reveals the user's dictionary.

Brian: Only if you draw it to the canvas. Can you provide those things on another layer, tied to the canvas, but not written into image?

Philip: Autofill: Browsers when you click a form element, you get a dropdown with, say, your name and your alter-ego. When you hover and it previews, that's super sensitive and not visible to Javascript. So you draw those previews on top, like you draw selects. So that's a way out.

Brian: That's the answer. Suggetion is you can draw them, but only above the canvas.

Philip: Spell check markers would probably break layering, so I've chosen not to drawn them, but autofill is already drawn above the DOM. Tooltips too.

Xiaocheng: Timing of paint event: Explainer is pretty vague: says if anything changes. Some changes can be processed completely off the main thread. Wondering if these have to trigger paint, would this be not implementable in a UA that processes such changes entirely off the main thread?

Philip: Canvas is inherently main-thread, so everything drawn in html-in-canvas has to be tied to the main thread. Canvas apps are 100% main thread, and html-in-canvas can't fix that, so drawing there ties things to the main thread. We talked with other engines about whether they could implement this. Have a proposal and a follow-on to support threaded effects. Could go into detail. Current API shape is 100% main thread, like canvas.

Xiaocheng: Are off-main-thread APIs supported? E.g. Video playing in canvas, shoudl I fire a paint event every frame?

Philip: Yes.

Xiaocheng: Have to hook changes from different thread to main thread.

Philip: Also need this hook if you're drawing video into WebGL.

Dan: Confidential information: why can't the developer opt into tainting, e.g. no read-back?

Philip: Can't do it because in the interactive case, need to draw into canvas and be able to tell JS when to redraw. Invalidation. And that can't support tainting. If we draw an input box and draw squiggles. Can't tell JS to redraw without revealing the thing we don't want to tell them. Interactivity blocks tainting. What if we allow cross-origin iframes but didn't have onpaint? They'd change but wouldn't appear changed.

Jeffrey: If canvas acquired someday an async interface, could this start supporting off-main-thread html-in-canvas?

Philip: Spent so much time for threaded scrolling; sucks to throw that away. Say the 2d canvas recorded everything it drew, and when you get to a scroller, it could replay the canvas on another thread. We think there's a path, but it's vaporware.

Xiaocheng: Thank you for coming. I'm very excited about the proposal, but have to play adversary here.

Philip: Thanks for looking closely.