#636: Pickling for Async Clipboard API

Visit on Github.

Opened May 13, 2021

Ya ya yawm TAG!

I'm requesting a TAG review of Pickling for Async Clipboard API.

Powerful web applications would like to exchange data payloads with web and native applications via the OS clipboard (copy-paste). The existing Web Platform has an API that supports the most popular standardized data types (text, image, rich text) across all platforms. However, this API does not scale to the long tail of specialized formats. In particular, custom formats, non-web-standard formats like TIFF (a large image format), and proprietary formats like .docx (a document format), are not supported by the current Web Platform. Pickling for Async Clipboard API aims to provide a solution to this problem, by letting web applications read and write custom, unsanitized, web-originated payloads using a standardized pickling format.

Further details:

  • I have reviewed the TAG's Web Platform Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): Editing TF
  • The group where standardization of this work is intended to be done ("unknown" if not known): Editing TF
  • Existing major pieces of multi-stakeholder review or discussion of this design: N/A
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Microsoft

You should also know that...

There have been discussions on Pickling Clipboard API during TPAC and also while discussing Raw Clipboard API. Slides: https://docs.google.com/presentation/d/1_fAgL54D0whQ497G8iL0K2kKpxiWDr3M7gXXSIS76II Editing TF Minutes: https://lists.w3.org/Archives/Public/public-editing-tf/2020Oct/0017.html

We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify @snianu @dway123 @BoCupp-Microsoft

Discussions

Discussed May 24, 2021 (See Github)

Hadley: do we need a design principle about using metaphors? E.g. Toast.

[discussion on use of pickling]

Dan: do we want to bikeshed?

Ken: pickling takes time - i hope this will be a quick operation?

Hadley: we should because we should make it clear that people outside their community will need to use this.

Ken: one concern - say a web site creates a custom format - then other web sites start to adopt this - then it becomes widespread without santization. Is this being restricted to a domain name? People reverse-engineer formats.

Dan: could lead to a lot of unsanitized content in clipboards...

Ken: right.

Lea: if the custom format gets sanitized it would have a different mime type -

Ken: one of the goals is to not sanitized...

Lea: they would have to change their code if the format ever became standardised

Ken: but could become a defacto standard... That's a risk because it becomes ... Example: I'm figma and I allow people to paste javascript because it's in my format. Now someone creates a figma clone. That adds malicious javascript which will be executed in figma if people place it in there. But if it comes from a different site and there's no way to see where it originates.. needs to be sanitized...

Lea: a little unclear how someone opts in to the pickling... is it based on name?

Ken: they do assume that native apps will support these formats as well.. so that is also a risk.

Dan: the risk is executtable code?

Ken: or triggering code paths that haven't been [locked down]...

Dan: some worrying stuff here about the clipboard event API... Looks like this issue is still open... not sure what the current state of things is here... Not sure what to do.

Ken:

Comment by @kenchris May 25, 2021 (See Github)

Hi there, @torgo, @LeaVerou and I looked at this today.

One concern - say a web site creates a custom format - then other web sites start to adopt this - then it becomes widespread without sanitization. Is this being restricted to a domain name? People reverse-engineer formats, and this could lead to a lot of unsanitized content in clipboards.

Example, I am Figma and I allow people to paste say JavaScript because it's in my format. Now someone creates an app that can read and write this format and provide some features not in Figma. This could lead users to use both apps and copy/paste between them. But maybe the second app adds malicious JS (or just triggers an untested code path in Figma) which will be executed in figma if people paste it in there.

There doesn't even seem to be a way for sites to see where the pasted content originates (like origin) so do their own sanitation.

Comment by @torgo May 25, 2021 (See Github)

Additional comments:

On naming - we've reviewed what you've written in the explainer, but we still feel there could be a lot of confusion about this API especially with non-english speakers. Not sure if there is anything that can be done here.

On privacy - good to see an additional user activation requirement. However, I was under the impression that there is a requirement for user activation on any clipboard read. Is this not the case?

Comment by @snianu May 27, 2021 (See Github)

Is this being restricted to a domain name?

We don't want to restrict to a domain because we want native apps to be able to consume contents from sites that opted-in to be using pickling formats, and vice versa. We expect, sites like Figma, GoogleDocs, Office and Libre will need to have access to it, to provide better user experience. Browsers run with the assumption that clipboard contents are untrusted, hence the sanitization of web-exposed formats. As far as pickling formats go, web apps should not assume the content is trusted either and do the due diligence themselves. And this is by design since we want to preserve as much of the original content as possible.

On naming

Pickling formats is not exposed as an API name anywhere. Are you referring to the usage in the explainer and the spec? We could also refer to it as Custom unsanitized mime types with a common naming pattern across browsers"? In the API direct option in the ClipboardItemOptions refers to the custom formats that shouldn't be sanitized. Are we OK with this naming?

On privacy

In Chromium, we do have user activation requirement for execCommand("copy") but I don't think we have one for async clipboard APIs read/write. We only have permission prompts for async clipboard read, but not write. For Pickling APIs we do want to add both user activation and permission prompt for read & write. However there were feedbacks from web developers that if it's gated behind user gesture requirement, then it would be too restrictive and fetching payload (which might be huge) from the server and writing into the clipboard might cause performance issues during copying/pasting. This is sort of implying that the operation is synchronous which defeats the purpose of async clipboard apis.

Adding @dway123 @whsieh @gked @BoCupp-Microsoft to comment more on this.

Comment by @torgo Jun 2, 2021 (See Github)

However there were feedbacks from web developers that if it's gated behind user gesture requirement, then it would be too restrictive and fetching payload (which might be huge) from the server and writing into the clipboard might cause performance issues during copying/pasting.

Is this still under discussion somewhere? I don't understand how this can be reconciled with the need to keep users' clipboard contents private when they visit a web page (which is related to one of our key design principles : https://www.w3.org/TR/design-principles/#safe-to-browse that it must be safe to visit a web page).

Comment by @gked Jun 2, 2021 (See Github)

However there were feedbacks from web developers that if it's gated behind user gesture requirement, then it would be too restrictive and fetching payload (which might be huge) from the server and writing into the clipboard might cause performance issues during copying/pasting.

Is this still under discussion somewhere? I don't understand how this can be reconciled with the need to keep users' clipboard contents private when they visit a web page (which is related to one of our key design principles : https://www.w3.org/TR/design-principles/#safe-to-browse that it must be safe to visit a web page).

There was a discussion on github which can be found here. To be clear, it is not just a random site accessing clipboard content. User will need to give clipboard access to the site through permissions API first. In addition, having a sticky user activation requirement (instead of transient) and an active document in focus would ensure that the webpage doesn't have access to clipboard without user explicitly interacting with the web page at least once.

We are reaching out to web applications so they can weigh-in on this bit as well.

CCing @garykac as FYI

Comment by @AbhishekR-Microsoft Jun 3, 2021 (See Github)

As far as pickling formats go, web apps should not assume the content is trusted either and do the due diligence themselves. And this is by design since we want to preserve as much of the original content as possible.

I am from Office team. Custom formats in Clipboard will benefit our users since it will drastically improve native to web apps copy paste fidelity.

We are good with doing sanitization of custom formats at our end.

Discussed Jun 7, 2021 (See Github)

Dan: discussion about other issues on previous clipboard APIs on privacy and user gesture. Question I still have is the statement from ?? seems to be that the discussion to mitigate against random sites accessing clipboard content users need to user permissions api, and a 'sticky' user activation requirement..

Tess: once the page gets a sticky user activation gesture it retains it so in the future when the user activation check happens it checks if it ever happened. Transient means at the time of the check it has to have one. I don't think that's good enough here. It does not define the nature of the user gesture that is causing the user activation to be the case. Just tapping on a page is not sufficient - what we wanted is for the user to have explicitly done something which maybe OS depending, that is explicitly a paste gesture. That might be cmd+v or tapping in a textarea on a mobile device and picking paste from the pill menu. It would actually be a paste. Tapping on the page for some other reason, like as a prelude to scrolling the page isn't sufficient for the user to say the page can know about the pasteboard.

Dan: Would changing it to transient fix the issue?

Tess: no. I don't believe user activation stuff in HTML spec contains a notion of a restricted form of user activation that carries some kind of intent. If they want to add this feature to the web they have to add that too.

Hadley: they were talking about the permissions API?

Tess: that's an upfront prompt

Hadley: what they've pointed to in the html spec is the kind of activation that means that... to prevent abuse of certain apis that could be annoying... allowed when user interacted with page at least once. Possible that sticky activation could happen for a site in the background?

Tess: I don't remember

Dan: eric is saying it nees to be active tab, they've taken that into account. STill feel like you could imagine a situation where you're phised to a site pretending to be your bank, it games your permission activation, it's the active tab and your user gesture has been satisified because through a dark pattern it tricks you into clicking the permission prompt, then it has access to your clipboard

Tess: it would take two tabs. Agreeing to the permission prompt doesn't provide user activation to the page. But there are a number of ways to induce you to tap on the page after you've accepted the permission. It's not clear to me that browsers would have an explicit permisson prompt here. If you look at existing platforms and the paste behaviour my understanding is that browsers have the freedom to always grant or always deny without bugging the user about it.. They put it behind permission api but that doesn't tell us if the user has explicitly agreed to anything. The only explicit signal from the user that they want to share the contents of the pasteboard is when they do a paste command - that's the vital thing. THe browser has to know they intend to paste. Pulling up edit and clicking paste, or pressing paste keyboard shortcut, or equivalent touch gesture

Hadley: why do we need a web api if those things are already in the OS?

Tess: there might not be a focusable field on the page in which you'r epasting. CUrrently if you focus an input type text and you pick paste the page doesn't get to know that was from yoru clipboard, it's just a new value for the input.

Dan: pasting pickled graphics formats onto a blank canvas you're not pasting text into a text field, it's complex into an application

Tess: exactly. No affordance for that. The page has to ask for the contents of the pasteboard. Common case is taken care of already.

Dan: when they say performance issue, they want that active tab to be able to start uploading a large piece of graphic ot the destination site... I dunno.. .maybe a way for these large things to be held in some kind of escrow buffer until there's an affirmative activation by the user?

`Replaying our discussion in today's TAG plenary call where we discussed this issue. The TAG consensus is that the mitigation described - permission, sticky activation, active document - is not sufficient. This is because it still could lead to unintentional leakage of the information held in the user's clipboard. We don't believe that the user activation as specified contains a notion of a restricted form of activation that carries some kind of intent (a specific intent to paste content). And if this feature should be added to the web, then that kind of activation feature (with intention) should also be added.

One scenario is: you're phished to a page that looks like your bank's web site. that page induces you to agree to the permission through a dark pattern (and also induces some additional interaction in the same way), and then (without an explicit paste command) the web page in question has access to the contents of your clipboard.

Another scenario: you're on a legitimate site and you explicitly paste something into a page which has permission to access your clipboard. Now you go off and copy something else (say a password) into your clipboard which is unrelated to your interaction with this site. And bring the browser window back up. Assuming this site is the active tab, this site would have access to your sensitive information on the clipboard.

We understand the performance issue that's been articulated. Is there another way to address this issue that can afford the user the protection we think they need here.`

Rossen: is your scenario that if I activate access to clipboard from site A, however I obtained the access to the clipboard, I'll be able to have acces to everything on the clipboard from what was put there by site B?

Dan: or by anything.. say you have a password copied from somewhere else, or a URL to a private document. Site A would have access to that information without you having used an explicit user activation.

Hadley: the user action doesn't necessarily means the user understands or thinks they're making happen whath is happening

Tess: only certain user gestures mean I'm trying to paste something

Rossen: the edit command is activate by something - gesture, keyboard, something else - is the trigger. Not necessarily ..

Dan: yes, but they're saying that that wouldn't be the case. Site A would have access to clipboard simply by you having clicked anywhere on the page after you have given permission. You've accepted the permission prompt, the tab is active, and you'v eclicked somewhere on the page. Doesn't need to be explicitly I want to paste a thing. Simply 'click okay to prove you're not a robot'. Now all the conditions are met for that site to have access to the contents of your clipboard. Tess is saying no, the user needs to explicitly say I want to paste.

Hadley: another scenario - you go to a site that is operating in good faith, you give it permission, you paste in whatever you intended to paste, then go off and do something else including copying in other sensitive information, then go back to that site. Then that site has access to the stuff you've copied in for something else, because it's sticky. That's not what a user would expect.

Amy: what if you've changed your mind after you've given permission? You go to the site and accept the permission because maybe you want to paste, do some other stuff on the site (interactions), and don't end up pasting. It would still have access to the clipboard then.

Comment by @torgo Jun 7, 2021 (See Github)

@gked

To be clear, it is not just a random site accessing clipboard content. User will need to give clipboard access to the site through permissions API first.

I'm not sure I'm convinced that putting something behind a permission by itself is sufficient for something this powerful and potential privacy infringing, especially considering how we have seen egregious gaming of privacy prompts by bad actors (see here: https://github.com/w3ctag/design-reviews/issues/337#issuecomment-561368571). However, adding a user activation requirement and being the active document in focus (as you've described) may provide additional mitigation. I put this on the agenda for this week's TAG calls to discuss further. Are the risks and mitigations documented in the appropriate security & privacy considerations section? If so, can you point me that way?

Comment by @gked Jun 7, 2021 (See Github)

@gked

To be clear, it is not just a random site accessing clipboard content. User will need to give clipboard access to the site through permissions API first.

I'm not sure I'm convinced that putting something behind a permission by itself is sufficient for something this powerful and potential privacy infringing, especially considering how we have seen egregious gaming of privacy prompts by bad actors (see here: #337 (comment)). However, adding a user activation requirement and being the active document in focus (as you've described) may provide additional mitigation. I put this on the agenda for this week's TAG calls to discuss further. Are the risks and mitigations documented in the appropriate security & privacy considerations section? If so, can you point me that way?

Thank you, looking forward to getting the feedback on this proposal. To your last question, I believe, @snianu has documented privacy and security risks before. Hey @snianu, could you please point @torgo to it?

Comment by @snianu Jun 8, 2021 (See Github)

Here is the link to the security/privacy section in the explainer: https://github.com/dway123/clipboard-pickling/blob/main/explainer.md#privacy-and-security

Comment by @torgo Jun 9, 2021 (See Github)

@gked :

Replaying our discussion in today's TAG plenary call where we discussed this issue. The TAG consensus is that the mitigation described - permission, sticky activation, active document - is not sufficient. This is because it still could lead to unintentional leakage of the information held in the user's clipboard. We don't believe that the user activation as specified contains a notion of a restricted form of activation that carries some kind of intent (a specific intent to paste content). And if this feature should be added to the web, then that kind of activation feature (with intention) should also be added.

One scenario is: you're phished to a page that looks like your bank's web site. that page induces you to agree to the permission through a dark pattern (and also induces some additional interaction in the same way), and then (without an explicit paste command) the web page in question has access to the contents of your clipboard.

Another scenario: you're on a legitimate site and you explicitly paste something into a page which has permission to access your clipboard. Now you go off and copy something else (say a password) into your clipboard which is unrelated to your interaction with this site. And bring the browser window back up. Assuming this site is the active tab, this site would have access to your sensitive information on the clipboard.

We understand the performance issue that's been articulated. Is there another way to address this issue that can afford the user the protection we think they need here.

Comment by @gked Jun 12, 2021 (See Github)

Thanks, @torgo for sharing TAG consensus!

Correct me if I am wrong but its sounds like security scenarios presented are applicable to not just pickling but any format used by async clipboard api. Pickling format in itself doesn't create any additional vectors of attack in this case.

We are reaching out to security reviewers that have reviewed async clipboard api before it was shipped, to better understand rational behind the current API design.

P.S. We took your feedback about "sticky" activation and moved to requiring "transient" activation for reading and writing to the clipboard.

Comment by @snianu Jun 28, 2021 (See Github)

Just a small update. We addressed few security concerns raised by internal security reviews as described below:

  1. Changed direct keyword to unsanitized to be more explicit about the content being read/written.
  2. Added transient user activation for reading and writing into the clipboard.
  3. Format names are also mangled while writing into the clipboard so it's clear to the consumer of those formats that the content is unsanitized and written from the browser. e.g. format names are prefixed with "Web" on Windows

The explainer has been updated and moved to editing working group repo: https://github.com/w3c/editing/blob/gh-pages/docs/clipboard-pickling/explainer.md#pickling-for-async-clipboard-api

Comment by @torgo Jul 7, 2021 (See Github)

@gked sorry for the delayed response but yes that's correct. the scenarios described are applicable to any format used by async clipboard. We did review async clipboard and we did raise this issue however it was our understanding that the feedback had been listened to. And indeed it looks to me from here https://github.com/w3c/clipboard-apis/issues/52 that this issue is still open. Will pick up on that issue.

Comment by @snianu Aug 17, 2021 (See Github)

@torgo Just checking if TAG got a chance to discuss this API. We made some changes in the format naming and how the custom formats are read/written via async read/write methods and would love to get some feedback on this.

Discussed Sep 1, 2021 (See Github)

Dan: they made some changes in format naming and how custom formats are read/written and want feedback

Dan: still the basic issue with the async clipboard api as speced doesn't require a user gesture to read the clipboard - the issue is still oepn but other documents treat it as resolved.

Sangwhan: it's hard to resolve - web ui component that makes a menu that tricks them to give access to the clipboard. If you restrict it to ^c, ^c equiv - it's ok. Once you self define your ui then all bets are off.

Dan: Safar chose to go with the ^c ^v equiv option... just the native system shortcuts.

Sangwhan: you could put the user gesture in the accept cookies dialog and people will just click yes and the site gets access to the clipboard. If you restrict to just the native system context menu system shortcuts that's fine but it alienates the mouse clicking user who has to go through the menu to find copy and paste

Dan: we discussed with Alex Russell, it leads to a situation where it's possible you could have a password or other sensitiv einformation sitting on your clipboard and you bring up your browser, navigate to your tab, no user gesture, and that tab has access to the contents of your clipboard and I don't think that's acceptable.

Sangwhan: if you have given permission

Dan: if you've given permission previously. Not a user expectation that if you've give permission previously.. I've given permission to paste things into MSWord, doesn't mean I think MSWord is going to snoop on my clipboard. People have lots of private data in their clipboard. This is a separate issue, but I feel like we left a comment which is that there are abuse scenarios not being given enough weight. This is the problem - this issue is open - but people are saying it's already decidedin the async clipboard api - if it's decided, why is the issue open? It's been two years.

Sangwhan: agree with the principle and the risks. I don't have a solution.

Dan: it's a timebomb until somebody .. this is a huge door open for websites to snoop on your clipboard

Lea: I agree.. but it doesn't seem to be related to the pickling proposal

Dan: the issue is that this proposal calls out the fact that pickled contents of clipboards can be more damaging, more snesitive, but doesn't specifically says they can't/won't implement any additional gesture requirements or UI requirements because it's dealt with in the async clipboard api. They said because async clipboard doesn't do it we're not going to do it.

Lea: I thought the examples about private data being leaked would not require pickling

Dan: what if you had an image of your child on your clipboard?

Lea: what happens today without pickling?

Sangwhan: image works fine right now. You don't need pickling for image support.

Lea: many applications already allow this

Sangwhan: if you do something like copy something from Sketch, a bunch of objects, then paste in the web, that becomes garbage. Either rasterizes or do nothing, or post garbage on the screen.

Lea: if there's any sensitive data in what you're copying it will still be leaked if you try to paste it today..

Sangwhan: with the current status you can get away with that as well

Lea: users try to paste whatever is pasteable, they don't know what the platform supports or what the web app supports

Sangwhan: even without async clipboard you could still access the clipboard, and that would mean you can access the problematic data. This has been here for a while. Once you have clipboards, unless you say we have a standardised way of saying all password like things or sensitive data has to be serialized in this way so every browser implementation can not leak it.. that's going to be a massive project

Dan: we should focus on the pickle aspect of the proposal, and in our closing comment we can say the clipboard access is still problematic, though we said that before and nothing happened

Sangwhan: yes. One thing pickling does let you do is you can put in carefully crafted payloads to blow native apps up. Which you couldn't do in the past. If you have.. given you can make someone click on an accept all cookies button, even if you gate writing to the clipboard behind a user interaction, if you know someone uses MSWord 2015.. you could craft a malicious payload that is targeted for MSWord 2015, hide behind accept all cookies dialog, when that user interaction happens write it to the clipboard, and hope they paste into MSWord2015 and it crashes and you get native execution.. this is hypothetical.

... Other question is how much stuff can you pickle into the clipboard? Could also use it as DoS if there's no upper limit. Here's 10gb of data into the clipboard..

Lea: can't you do that already?

Sangwhan: you could, I think...?

Lea: stuffing extra data in what they copied.. have you copied data from an article to find it has more sentences tucked into it? If they wanted to do DoS that way they could stuff in gb of data.

Sangwhan: you could just write 2 million characters in the DOM. But I think the deserialization delayed bomb is probably somethign theoretically possible. The thing is that right now images are sanitized when you copy them into the clipboard. Web images are sanitized by the browser in a way that it takes away some of the dangerous bits (but also loses some information, eg. colour spaces). The reason is because of the attack surfaces associated with that. Once you are allowing aribtrary deserializable data into the clipboard you have the problem that you don't know how to sanitize because you don't know how to deserialize. That's the risk that comes with this. We can't say we will never allow anything complex because then the web can't do complex stuff. Eg. you can't make autocad for web because we only allow rich text and images.

Lea: do we have something to suggest to make this more secure?

Sangwhan: I don't think so.. they spend all their time thining about this problem probably becuase it's the most risky aspect of this feature. They do bring this up in this section onwards. They have thought about this. I personally say it's worth an origin trial to see how the real world implications and applications can make use of it and what kind of limitations are hit. I'm sure there are plenty of things that have to be ironed out. I'm okay with this proposal, I'm sure there is some extra stuff that will need to be considered once we try to integrate this into an existing application, eg. something like Figma. What happens when you copy from Figma to illustrator? From Illustrator to Figma? Sketch to Figma?

Dan: I think tha'ts out of scope.. you're talking about something needed to be implemented by figma..

Sangwhan: more like can figma actually implement deserialzing something copied out of Sketch. This is a whole new beast.

Lea: this is the primary use case? being able to copy data like this from applications to web apps and vice versa?

Sangwhan: yeah. It would be interesting to see the origin trial and the feedback from different services interested in this.

Lea: we should probably phrase it in a way that is not chrome specific?

Sangwhan: makes sense

Dan: can just say trial

Comment by @cynthia Sep 14, 2021 (See Github)

Apologies for the long delay! We discussed this at length during our VF2F, and while we think this would be a great addition to the platform, it's also a scary+open ended feature. The design itself I think makes a lot of sense and we are happy with it, but we'd like to see this being tried out in the wild (say, with some early adopters) and getting their feedback would be useful to know if there are details/features that we potentially missed in the initial design.

Do you think it would make sense to give it a trial run in the wild, and re-open after seeing what kind of feedback comes back? (presumably, also with updates to the design) @snianu

Comment by @snianu Sep 15, 2021 (See Github)

Thank you @cynthia for the feedback! Yes, we do have partners who are very much interested in using this API as it would unlock some of the powerful editing features in their sites(and interop with native apps) and also provide high fidelity copy-paste content in general. For experiments, we do need a resolution from TAG, so is it possible to get some resolution here? I'll reopen the issue & post any feedback we get from our partners if it leads to any significant design changes (probably unlikely).

Comment by @cynthia Sep 15, 2021 (See Github)

Done, please request a re-open when you have more data!