#925: Delayed Clipboard Rendering
Discussions
Discussed
Jan 15, 2024 (See Github)
Dan: this is an early review. They've said no considerable privacy concerns expected.
Yves: there is a contentious issue regarding privacy.. contradictory.
Dan: privacy issues for custom web formats. Should be drawn out in the explainer. Positive feedback from safari = still open standards position, neutral from firefox = open but not replied to standards position. You paste something, it causes the original application that copied it to do some work. Doesn't that feel like a potential security issue?
Peter: when dyou do the initial copying the source application puts data on the clipboard - it just doesn't have to put all of the formats it could potentially rpovode, just one and a list of other formats. At paste time, the receiving application says I want it in this format. Then the browser can go back to the original application and say convert this blob of data on the clipboard to this format.
Dan: what happenes when content is pasted from another tab?
Peter: you should be going back to the tab it was copied from
Dan: feels like two situations - copied from another application, and one copied from another tab in the same browser
Hadley: I'm reading that they're not paying attention to cross application use case
Peter: only copying from a web app, to anywhere
Hadley: given the clipboard is generally an operating system thing I think it has the potential to really confuse things for users. A use case is excel in the browser - when you copy an excel cell you can get a whole bunch of formatting and formula to go with the copying. Why that would work from excel in the browser and not excel the application would be really confusing
Peter: this should only impact things copied from a web app. Excel the app already behaves this way. If I have an excel app open and I copy the cells, the only thing on the clipboard is some excel native binary data. It's at paste time that the browser goes back and asks for the format to change..
Dan: scenario says user copies cells from native excel app and pastes it into a web based..
Hadley: in the last sentence of that section with delayed clipboard rendering excel online app is looking to handle custom formats
Peter: if you're copying from a native app, then the OS goes back to the native app to translate the clipboard, this API is not involved. This API is only involved with copying from a webapp. Concerned about - if I copied from an excel online source, and paste into something else, then I'm going to take that data and send it back to the excel online server to translate. You have extra network overhead and serverside processing. I don't think it's extra, if you didn't do this the server would have had to generate all the different formats at a copy time - still saving bandwidth, just delaying and filtering down to the one you need, if you even need it. You can copy things and never paste them. You'd never paste most of the formats that excel can translate into.
Hadley: true
Peter: conceptually I think it's fine. Not sure I see the privacy issue.
Dan: writes comment
Comment by @torgo Jan 15, 2024 (See Github)
Hi @snianu considering you've highlighted a potential issue around privacy - it feels like that should be drawn out in the explainer? It's good to see some positive signals from another implementer. We're a bit confused whether this proposal is intending to cover the cases where something has been coped from a native app and pasted into a web app, or the other way around, or whether it's intended only for copying from one web app to another web app. We think it might be only for web-to-web or web-to-native - is that correct? Can you clarify?
Comment by @snianu Jan 23, 2024 (See Github)
@torgo Sorry for the delayed response.
considering you've highlighted a potential https://github.com/w3c/editing/issues/439 - it feels like that should be drawn out in the explainer?
Sounds good. I'll add this issue to the explainer.
We think it might be only for web-to-web or web-to-native - is that correct? Can you clarify?
It is only applicable if copy
(not paste) is being performed in a web app. The destination app for paste doesn't matter as the system clipboard would call back into the source app during the paste operation to read the data for the requested format.
Comment by @hober Jan 24, 2024 (See Github)
I left this comment: https://github.com/MicrosoftEdge/MSEdgeExplainers/pull/742/files#r1464976569 I then got pointed to w3c/editing#439
Discussed
Feb 5, 2024 (See Github)
The below was filed as w3c/editing#459
:
@plinss, @hober, and I discussed the feature. This seems fairly reasonable on the surface, but https://github.com/w3c/editing/issues/439 is particularly concerning.
It seems like there is an inherent privacy issue here in that the target application reveals something about itself when pasted into. The suggestions that have been made in the issue, which involve resolving the clipboard items on a timer, would seem to undermine the key advantage of the deferral and do not provide protection within the timer.
The key thing to realize is that the clipboard is a communication medium between the copied content and the application that receives a paste. This is unavoidable given our current model where a website can put whatever it pleases on the clipboard (with the only condition being that they receive an interaction of some sort). That communication is currently one-way and websites don't necessarily get to know about the destination or control it. That makes the channel less effective as a means of learning things about people. The delayed rendering creates a potential bidirectional channel, with the destination choosing from a multiple choice selection. The site learns what choice is made. The choice itself carries novel private information. The timing of the paste is also revealed.
There are other approaches:
- Render all formats when paste occurs. This still reveals timing, but would not reveal the choice of format.
- A variant of this would be to generate the requested format and a randomized subset of the other formats (differential privacy)
- At copy time, produce a single format from which all others can be produced. Let the destination application perform format translation.
- As an option, if the single format is not supported at the destination, sites could provide a worklet that can perform translation into different formats. This worklet would be denied access to any communication, so the source site would not have any means of learning the choice of format.
- Finally, we could define a new media type that carries a URL. That URL, when resolved, provides the destination application with information in whatever form it desires.
Comment by @snianu Feb 8, 2024 (See Github)
Discussed with the EditingWG and we came to a consensus on the privacy issue that we will be supporting delayed clipboard rendering for just the built-in formats that aren't tied to any app ecosystem. Please see https://github.com/w3c/editing/issues/439#issuecomment-1934719705 for more info. Thanks again for the feedback!
Discussed
Feb 12, 2024 (See Github)
typedef (DOMString or Blob) ClipboardItemValue; // should this be `Promise<ClipboardItemValue>` instead?
callback ClipboardItemValueCallback = ClipboardItemValue(); // this should take a `DOMString type` argument so you don't need a closure always
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData; // A promise here is a bit strange. I might accept that you have three things: (DOMString or Blob), Promise<(DOMString or Blob)>, or a callback.
[SecureContext, Exposed=Window] // should this be transferrable?
interface ClipboardItem {
constructor(record<DOMString, ClipboardItemData> items,
optional ClipboardItemOptions options = {});
readonly attribute PresentationStyle presentationStyle; // what populates this? options?
readonly attribute FrozenArray<DOMString> types;
Promise<Blob> getType(DOMString type); // maybe just `get(DOMString type)`
};
Comment by @plinss Feb 12, 2024 (See Github)
Discussed during a breakout today with @hober and @martinthomson. We're happy to see the privacy concerns addressed (though we do see this approach as somewhat kicking the can down the road if more formats are exposed later).
We do have some concerns about the API shape:
typedef (DOMString or Blob) ClipboardItemValue; // should this be `Promise<ClipboardItemValue>` instead?
callback ClipboardItemValueCallback = ClipboardItemValue(); // this should take a `DOMString type` argument so you don't need a closure always
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData; // A promise here is a bit strange. I might accept that you have three things: (DOMString or Blob), Promise<(DOMString or Blob)>, or a callback.
[SecureContext, Exposed=Window] // should this be transferrable?
interface ClipboardItem {
constructor(record<DOMString, ClipboardItemData> items,
optional ClipboardItemOptions options = {});
readonly attribute PresentationStyle presentationStyle; // what populates this? options?
readonly attribute FrozenArray<DOMString> types;
Promise<Blob> getType(DOMString type); // maybe just `get(DOMString type)`
};
Comment by @snianu Feb 12, 2024 (See Github)
Just wanted to note that the only changes we're making in ClipboardItem
interface IDL for delayed clipboard rendering is the addition of a callback:
callback ClipboardItemValueCallback = ClipboardItemValue();
Many of the IDL definitions for ClipboardItem
interface are existing changes: https://w3c.github.io/clipboard-apis/#clipboard-item-interface
typedef (DOMString or Blob) ClipboardItemValue; // should this be
Promise<ClipboardItemValue>
instead?
The reason why we don't have a Promise here is because IDL compiler doesn't support union of promises. Here is the relevant issue for this: https://github.com/whatwg/webidl/issues/1278#issuecomment-1461691793
Comment by @plinss Feb 13, 2024 (See Github)
If that's the case, then why do you even have to add a callback at all? Couldn't the author simply pass an unresolved promise to the ClipboardItem constructor and the promise gets resolved at paste time?
e.g.:
async generateExpensiveBlob(): Promise<Blob> {
return new Blob(...);
}
const clipboardItemInput = new ClipboardItem({
'text/plain': generateExpensiveBlob(),
});
navigator.clipboard.write([clipboardItemInput]);
Gets you delayed rendering, no?
Comment by @snianu Feb 13, 2024 (See Github)
async generateExpensiveBlob(): Promise<Blob> { return new Blob(...); }
The primary purpose of delayed rendering is to not produce the data at all for a format that is never used on paste. In this case, the web authors have to be ready with the data when the promise executor function is run on write, correct?
Here is some discussion related to it: https://github.com/w3c/editing/issues/417#issuecomment-1461652377
Comment by @plinss Feb 13, 2024 (See Github)
Right, never mind me, I was confusing this with something else (working on too many projects these days).
Note though that we did recommend that the callback gets the type passed to it so that the a single callback could handle multiple types without creating extra closures.
Also during our discussion, were were questioning if the callback itself should return a promise (to make writing callbacks that need to do async processing easier). At the least the result of the callback should be awaited to make a promise return possible.
Discussed
Feb 26, 2024 (See Github)
Resolved: satisfied with concerns
To conclude here, as this is an early review, we're OK with the resolution of the capability-probing thing (deferral) and are content with the explanation about the use of promises (our bad for missing that). We do think that there are opportunities missed -- in particular, the callback could return a promise also -- but don't see any concrete problems.
Comment by @martinthomson Feb 26, 2024 (See Github)
To conclude here, as this is an early review, we're OK with the resolution of the capability-probing thing (deferral) and are content with the explanation about the use of promises (our bad for missing that). We do think that there are opportunities missed -- in particular, the callback could return a promise also -- but don't see any concrete problems.
OpenedJan 3, 2024
こんにちは TAG-さん!
I'm requesting a TAG review of Delayed Clipboard Rendering.
Delayed clipboard rendering is the ability to delay the generation of a particular payload until it is needed by the target applications. It is useful when source applications support several clipboard formats, some or all of which are time-consuming to render.
Further details:
You should also know that...
Existing discussions on the API shape: https://github.com/w3c/editing/issues/423 Privacy issue for web custom formats: https://github.com/w3c/editing/issues/439 General discussions: https://github.com/w3c/editing/issues/417
We'd prefer the TAG provide feedback as (please delete all but the desired option):
🐛 open issues in our GitHub repo for each point of feedback