#636: Pickling for Async Clipboard API

Visit on Github.

Opened May 13, 2021

Ya ya yawm TAG!

I'm requesting a TAG review of Pickling for Async Clipboard API.

Powerful web applications would like to exchange data payloads with web and native applications via the OS clipboard (copy-paste). The existing Web Platform has an API that supports the most popular standardized data types (text, image, rich text) across all platforms. However, this API does not scale to the long tail of specialized formats. In particular, custom formats, non-web-standard formats like TIFF (a large image format), and proprietary formats like .docx (a document format), are not supported by the current Web Platform. Pickling for Async Clipboard API aims to provide a solution to this problem, by letting web applications read and write custom, unsanitized, web-originated payloads using a standardized pickling format.

Further details:

  • I have reviewed the TAG's Web Platform Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): Editing TF
  • The group where standardization of this work is intended to be done ("unknown" if not known): Editing TF
  • Existing major pieces of multi-stakeholder review or discussion of this design: N/A
  • Major unresolved issues with or opposition to this design: N/A
  • This work is being funded by: Microsoft

You should also know that...

There have been discussions on Pickling Clipboard API during TPAC and also while discussing Raw Clipboard API. Slides: https://docs.google.com/presentation/d/1_fAgL54D0whQ497G8iL0K2kKpxiWDr3M7gXXSIS76II Editing TF Minutes: https://lists.w3.org/Archives/Public/public-editing-tf/2020Oct/0017.html

We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify @snianu @dway123 @BoCupp-Microsoft

Discussions

2021-05-24

Minutes

Hadley: do we need a design principle about using metaphors? E.g. Toast.

[discussion on use of pickling]

Dan: do we want to bikeshed?

Ken: pickling takes time - i hope this will be a quick operation?

Hadley: we should because we should make it clear that people outside their community will need to use this.

Ken: one concern - say a web site creates a custom format - then other web sites start to adopt this - then it becomes widespread without santization. Is this being restricted to a domain name? People reverse-engineer formats.

Dan: could lead to a lot of unsanitized content in clipboards...

Ken: right.

Lea: if the custom format gets sanitized it would have a different mime type -

Ken: one of the goals is to not sanitized...

Lea: they would have to change their code if the format ever became standardised

Ken: but could become a defacto standard... That's a risk because it becomes ... Example: I'm figma and I allow people to paste javascript because it's in my format. Now someone creates a figma clone. That adds malicious javascript which will be executed in figma if people place it in there. But if it comes from a different site and there's no way to see where it originates.. needs to be sanitized...

Lea: a little unclear how someone opts in to the pickling... is it based on name?

Ken: they do assume that native apps will support these formats as well.. so that is also a risk.

Dan: the risk is executtable code?

Ken: or triggering code paths that haven't been [locked down]...

Dan: some worrying stuff here about the clipboard event API... Looks like this issue is still open... not sure what the current state of things is here... Not sure what to do.

Ken:

2021-06-07

Minutes

Dan: discussion about other issues on previous clipboard APIs on privacy and user gesture. Question I still have is the statement from ?? seems to be that the discussion to mitigate against random sites accessing clipboard content users need to user permissions api, and a 'sticky' user activation requirement..

Tess: once the page gets a sticky user activation gesture it retains it so in the future when the user activation check happens it checks if it ever happened. Transient means at the time of the check it has to have one. I don't think that's good enough here. It does not define the nature of the user gesture that is causing the user activation to be the case. Just tapping on a page is not sufficient - what we wanted is for the user to have explicitly done something which maybe OS depending, that is explicitly a paste gesture. That might be cmd+v or tapping in a textarea on a mobile device and picking paste from the pill menu. It would actually be a paste. Tapping on the page for some other reason, like as a prelude to scrolling the page isn't sufficient for the user to say the page can know about the pasteboard.

Dan: Would changing it to transient fix the issue?

Tess: no. I don't believe user activation stuff in HTML spec contains a notion of a restricted form of user activation that carries some kind of intent. If they want to add this feature to the web they have to add that too.

Hadley: they were talking about the permissions API?

Tess: that's an upfront prompt

Hadley: what they've pointed to in the html spec is the kind of activation that means that... to prevent abuse of certain apis that could be annoying... allowed when user interacted with page at least once. Possible that sticky activation could happen for a site in the background?

Tess: I don't remember

Dan: eric is saying it nees to be active tab, they've taken that into account. STill feel like you could imagine a situation where you're phised to a site pretending to be your bank, it games your permission activation, it's the active tab and your user gesture has been satisified because through a dark pattern it tricks you into clicking the permission prompt, then it has access to your clipboard

Tess: it would take two tabs. Agreeing to the permission prompt doesn't provide user activation to the page. But there are a number of ways to induce you to tap on the page after you've accepted the permission. It's not clear to me that browsers would have an explicit permisson prompt here. If you look at existing platforms and the paste behaviour my understanding is that browsers have the freedom to always grant or always deny without bugging the user about it.. They put it behind permission api but that doesn't tell us if the user has explicitly agreed to anything. The only explicit signal from the user that they want to share the contents of the pasteboard is when they do a paste command - that's the vital thing. THe browser has to know they intend to paste. Pulling up edit and clicking paste, or pressing paste keyboard shortcut, or equivalent touch gesture

Hadley: why do we need a web api if those things are already in the OS?

Tess: there might not be a focusable field on the page in which you'r epasting. CUrrently if you focus an input type text and you pick paste the page doesn't get to know that was from yoru clipboard, it's just a new value for the input.

Dan: pasting pickled graphics formats onto a blank canvas you're not pasting text into a text field, it's complex into an application

Tess: exactly. No affordance for that. The page has to ask for the contents of the pasteboard. Common case is taken care of already.

Dan: when they say performance issue, they want that active tab to be able to start uploading a large piece of graphic ot the destination site... I dunno.. .maybe a way for these large things to be held in some kind of escrow buffer until there's an affirmative activation by the user?

`Replaying our discussion in today's TAG plenary call where we discussed this issue. The TAG consensus is that the mitigation described - permission, sticky activation, active document - is not sufficient. This is because it still could lead to unintentional leakage of the information held in the user's clipboard. We don't believe that the user activation as specified contains a notion of a restricted form of activation that carries some kind of intent (a specific intent to paste content). And if this feature should be added to the web, then that kind of activation feature (with intention) should also be added.

One scenario is: you're phished to a page that looks like your bank's web site. that page induces you to agree to the permission through a dark pattern (and also induces some additional interaction in the same way), and then (without an explicit paste command) the web page in question has access to the contents of your clipboard.

Another scenario: you're on a legitimate site and you explicitly paste something into a page which has permission to access your clipboard. Now you go off and copy something else (say a password) into your clipboard which is unrelated to your interaction with this site. And bring the browser window back up. Assuming this site is the active tab, this site would have access to your sensitive information on the clipboard.

We understand the performance issue that's been articulated. Is there another way to address this issue that can afford the user the protection we think they need here.`

Rossen: is your scenario that if I activate access to clipboard from site A, however I obtained the access to the clipboard, I'll be able to have acces to everything on the clipboard from what was put there by site B?

Dan: or by anything.. say you have a password copied from somewhere else, or a URL to a private document. Site A would have access to that information without you having used an explicit user activation.

Hadley: the user action doesn't necessarily means the user understands or thinks they're making happen whath is happening

Tess: only certain user gestures mean I'm trying to paste something

Rossen: the edit command is activate by something - gesture, keyboard, something else - is the trigger. Not necessarily ..

Dan: yes, but they're saying that that wouldn't be the case. Site A would have access to clipboard simply by you having clicked anywhere on the page after you have given permission. You've accepted the permission prompt, the tab is active, and you'v eclicked somewhere on the page. Doesn't need to be explicitly I want to paste a thing. Simply 'click okay to prove you're not a robot'. Now all the conditions are met for that site to have access to the contents of your clipboard. Tess is saying no, the user needs to explicitly say I want to paste.

Hadley: another scenario - you go to a site that is operating in good faith, you give it permission, you paste in whatever you intended to paste, then go off and do something else including copying in other sensitive information, then go back to that site. Then that site has access to the stuff you've copied in for something else, because it's sticky. That's not what a user would expect.

Amy: what if you've changed your mind after you've given permission? You go to the site and accept the permission because maybe you want to paste, do some other stuff on the site (interactions), and don't end up pasting. It would still have access to the clipboard then.

2021-09-Gethen

Minutes

Dan: they made some changes in format naming and how custom formats are read/written and want feedback

Dan: still the basic issue with the async clipboard api as speced doesn't require a user gesture to read the clipboard - the issue is still oepn but other documents treat it as resolved.

Sangwhan: it's hard to resolve - web ui component that makes a menu that tricks them to give access to the clipboard. If you restrict it to ^c, ^c equiv - it's ok. Once you self define your ui then all bets are off.

Dan: Safar chose to go with the ^c ^v equiv option... just the native system shortcuts.

Sangwhan: you could put the user gesture in the accept cookies dialog and people will just click yes and the site gets access to the clipboard. If you restrict to just the native system context menu system shortcuts that's fine but it alienates the mouse clicking user who has to go through the menu to find copy and paste

Dan: we discussed with Alex Russell, it leads to a situation where it's possible you could have a password or other sensitiv einformation sitting on your clipboard and you bring up your browser, navigate to your tab, no user gesture, and that tab has access to the contents of your clipboard and I don't think that's acceptable.

Sangwhan: if you have given permission

Dan: if you've given permission previously. Not a user expectation that if you've give permission previously.. I've given permission to paste things into MSWord, doesn't mean I think MSWord is going to snoop on my clipboard. People have lots of private data in their clipboard. This is a separate issue, but I feel like we left a comment which is that there are abuse scenarios not being given enough weight. This is the problem - this issue is open - but people are saying it's already decidedin the async clipboard api - if it's decided, why is the issue open? It's been two years.

Sangwhan: agree with the principle and the risks. I don't have a solution.

Dan: it's a timebomb until somebody .. this is a huge door open for websites to snoop on your clipboard

Lea: I agree.. but it doesn't seem to be related to the pickling proposal

Dan: the issue is that this proposal calls out the fact that pickled contents of clipboards can be more damaging, more snesitive, but doesn't specifically says they can't/won't implement any additional gesture requirements or UI requirements because it's dealt with in the async clipboard api. They said because async clipboard doesn't do it we're not going to do it.

Lea: I thought the examples about private data being leaked would not require pickling

Dan: what if you had an image of your child on your clipboard?

Lea: what happens today without pickling?

Sangwhan: image works fine right now. You don't need pickling for image support.

Lea: many applications already allow this

Sangwhan: if you do something like copy something from Sketch, a bunch of objects, then paste in the web, that becomes garbage. Either rasterizes or do nothing, or post garbage on the screen.

Lea: if there's any sensitive data in what you're copying it will still be leaked if you try to paste it today..

Sangwhan: with the current status you can get away with that as well

Lea: users try to paste whatever is pasteable, they don't know what the platform supports or what the web app supports

Sangwhan: even without async clipboard you could still access the clipboard, and that would mean you can access the problematic data. This has been here for a while. Once you have clipboards, unless you say we have a standardised way of saying all password like things or sensitive data has to be serialized in this way so every browser implementation can not leak it.. that's going to be a massive project

Dan: we should focus on the pickle aspect of the proposal, and in our closing comment we can say the clipboard access is still problematic, though we said that before and nothing happened

Sangwhan: yes. One thing pickling does let you do is you can put in carefully crafted payloads to blow native apps up. Which you couldn't do in the past. If you have.. given you can make someone click on an accept all cookies button, even if you gate writing to the clipboard behind a user interaction, if you know someone uses MSWord 2015.. you could craft a malicious payload that is targeted for MSWord 2015, hide behind accept all cookies dialog, when that user interaction happens write it to the clipboard, and hope they paste into MSWord2015 and it crashes and you get native execution.. this is hypothetical.

... Other question is how much stuff can you pickle into the clipboard? Could also use it as DoS if there's no upper limit. Here's 10gb of data into the clipboard..

Lea: can't you do that already?

Sangwhan: you could, I think...?

Lea: stuffing extra data in what they copied.. have you copied data from an article to find it has more sentences tucked into it? If they wanted to do DoS that way they could stuff in gb of data.

Sangwhan: you could just write 2 million characters in the DOM. But I think the deserialization delayed bomb is probably somethign theoretically possible. The thing is that right now images are sanitized when you copy them into the clipboard. Web images are sanitized by the browser in a way that it takes away some of the dangerous bits (but also loses some information, eg. colour spaces). The reason is because of the attack surfaces associated with that. Once you are allowing aribtrary deserializable data into the clipboard you have the problem that you don't know how to sanitize because you don't know how to deserialize. That's the risk that comes with this. We can't say we will never allow anything complex because then the web can't do complex stuff. Eg. you can't make autocad for web because we only allow rich text and images.

Lea: do we have something to suggest to make this more secure?

Sangwhan: I don't think so.. they spend all their time thining about this problem probably becuase it's the most risky aspect of this feature. They do bring this up in this section onwards. They have thought about this. I personally say it's worth an origin trial to see how the real world implications and applications can make use of it and what kind of limitations are hit. I'm sure there are plenty of things that have to be ironed out. I'm okay with this proposal, I'm sure there is some extra stuff that will need to be considered once we try to integrate this into an existing application, eg. something like Figma. What happens when you copy from Figma to illustrator? From Illustrator to Figma? Sketch to Figma?

Dan: I think tha'ts out of scope.. you're talking about something needed to be implemented by figma..

Sangwhan: more like can figma actually implement deserialzing something copied out of Sketch. This is a whole new beast.

Lea: this is the primary use case? being able to copy data like this from applications to web apps and vice versa?

Sangwhan: yeah. It would be interesting to see the origin trial and the feedback from different services interested in this.

Lea: we should probably phrase it in a way that is not chrome specific?

Sangwhan: makes sense

Dan: can just say trial