#925: Delayed Clipboard Rendering

Visit on Github.

Opened Jan 3, 2024

こんにちは TAG-さん!

I'm requesting a TAG review of Delayed Clipboard Rendering.

Delayed clipboard rendering is the ability to delay the generation of a particular payload until it is needed by the target applications. It is useful when source applications support several clipboard formats, some or all of which are time-consuming to render.

Further details:

You should also know that...

Existing discussions on the API shape: https://github.com/w3c/editing/issues/423 Privacy issue for web custom formats: https://github.com/w3c/editing/issues/439 General discussions: https://github.com/w3c/editing/issues/417

We'd prefer the TAG provide feedback as (please delete all but the desired option):

🐛 open issues in our GitHub repo for each point of feedback

Discussions

2024-01-15

Minutes

Dan: this is an early review. They've said no considerable privacy concerns expected.

Yves: there is a contentious issue regarding privacy.. contradictory.

Dan: privacy issues for custom web formats. Should be drawn out in the explainer. Positive feedback from safari = still open standards position, neutral from firefox = open but not replied to standards position. You paste something, it causes the original application that copied it to do some work. Doesn't that feel like a potential security issue?

Peter: when dyou do the initial copying the source application puts data on the clipboard - it just doesn't have to put all of the formats it could potentially rpovode, just one and a list of other formats. At paste time, the receiving application says I want it in this format. Then the browser can go back to the original application and say convert this blob of data on the clipboard to this format.

Dan: what happenes when content is pasted from another tab?

Peter: you should be going back to the tab it was copied from

Dan: feels like two situations - copied from another application, and one copied from another tab in the same browser

Hadley: I'm reading that they're not paying attention to cross application use case

Peter: only copying from a web app, to anywhere

Hadley: given the clipboard is generally an operating system thing I think it has the potential to really confuse things for users. A use case is excel in the browser - when you copy an excel cell you can get a whole bunch of formatting and formula to go with the copying. Why that would work from excel in the browser and not excel the application would be really confusing

Peter: this should only impact things copied from a web app. Excel the app already behaves this way. If I have an excel app open and I copy the cells, the only thing on the clipboard is some excel native binary data. It's at paste time that the browser goes back and asks for the format to change..

Dan: scenario says user copies cells from native excel app and pastes it into a web based..

Hadley: in the last sentence of that section with delayed clipboard rendering excel online app is looking to handle custom formats

Peter: if you're copying from a native app, then the OS goes back to the native app to translate the clipboard, this API is not involved. This API is only involved with copying from a webapp. Concerned about - if I copied from an excel online source, and paste into something else, then I'm going to take that data and send it back to the excel online server to translate. You have extra network overhead and serverside processing. I don't think it's extra, if you didn't do this the server would have had to generate all the different formats at a copy time - still saving bandwidth, just delaying and filtering down to the one you need, if you even need it. You can copy things and never paste them. You'd never paste most of the formats that excel can translate into.

Hadley: true

Peter: conceptually I think it's fine. Not sure I see the privacy issue.

Dan: writes comment

2024-02-05

Minutes

The below was filed as w3c/editing#459:

@plinss, @hober, and I discussed the feature. This seems fairly reasonable on the surface, but https://github.com/w3c/editing/issues/439 is particularly concerning.

It seems like there is an inherent privacy issue here in that the target application reveals something about itself when pasted into. The suggestions that have been made in the issue, which involve resolving the clipboard items on a timer, would seem to undermine the key advantage of the deferral and do not provide protection within the timer.

The key thing to realize is that the clipboard is a communication medium between the copied content and the application that receives a paste. This is unavoidable given our current model where a website can put whatever it pleases on the clipboard (with the only condition being that they receive an interaction of some sort). That communication is currently one-way and websites don't necessarily get to know about the destination or control it. That makes the channel less effective as a means of learning things about people. The delayed rendering creates a potential bidirectional channel, with the destination choosing from a multiple choice selection. The site learns what choice is made. The choice itself carries novel private information. The timing of the paste is also revealed.

There are other approaches:

  1. Render all formats when paste occurs. This still reveals timing, but would not reveal the choice of format.
    1. A variant of this would be to generate the requested format and a randomized subset of the other formats (differential privacy)
  2. At copy time, produce a single format from which all others can be produced. Let the destination application perform format translation.
    1. As an option, if the single format is not supported at the destination, sites could provide a worklet that can perform translation into different formats. This worklet would be denied access to any communication, so the source site would not have any means of learning the choice of format.
  3. Finally, we could define a new media type that carries a URL. That URL, when resolved, provides the destination application with information in whatever form it desires.
2024-02-12

Minutes

typedef (DOMString or Blob) ClipboardItemValue; // should this be `Promise<ClipboardItemValue>` instead?
callback ClipboardItemValueCallback = ClipboardItemValue(); // this should take a `DOMString type` argument so you don't need a closure always
typedef Promise<(ClipboardItemValue or ClipboardItemValueCallback)> ClipboardItemData; // A promise here is a bit strange.  I might accept that you have three things: (DOMString or Blob), Promise<(DOMString or Blob)>, or a callback.

[SecureContext, Exposed=Window] // should this be transferrable?
interface ClipboardItem {
  constructor(record<DOMString, ClipboardItemData> items,
              optional ClipboardItemOptions options = {});

  readonly attribute PresentationStyle presentationStyle;  // what populates this? options?
  readonly attribute FrozenArray<DOMString> types;

  Promise<Blob> getType(DOMString type); // maybe just `get(DOMString type)`
};
2024-02-26

Minutes

Resolved: satisfied with concerns

To conclude here, as this is an early review, we're OK with the resolution of the capability-probing thing (deferral) and are content with the explanation about the use of promises (our bad for missing that). We do think that there are opportunities missed -- in particular, the callback could return a promise also -- but don't see any concrete problems.