#587: EyeDropper API

Visit on Github.

Opened Dec 16, 2020

HIQaH! QaH! TAG!

I'm requesting a TAG review of EyeDropper API.

The EyeDropper API enables developers to use a browser-supplied eyedropper in the construction of custom color pickers.

Further details:

  • I have reviewed the TAG's API Design Principles
  • The group where the incubation/design work on this is being done (or is intended to be done in the future): WICG
  • The group where standardization of this work is intended to be done ("unknown" if not known): Unknown
  • Existing major pieces of multi-stakeholder review or discussion of this design: Unknown
  • Major unresolved issues with or opposition to this design: None at the moment
  • This work is being funded by: Microsoft

We'd prefer the TAG provide feedback as (please delete all but the desired option): 🐛 open issues in our GitHub repo for each point of feedback

Discussions

2021-01-Kronos

Minutes

  • Missing a list of use cases

Lea: Top of mind for me is color format. We're not going to present all on-screen colors in implemented CSS>

... Bunch of color formats can go beyond sRGB gamut, but they're not implemented yet, and it seems that this API shouldn't block on those.

... However, only returning sRGB also seems bad.

... CSS color is lagging... hex colors have a specific meaning today, which is not device RGB but sRGB. They can't represent up to 1/3 of on-screen colors... most Apple devices, many other high end devices have (?) screens.

... Remaining issues are syntax/API issues.

... They modeled it after <input type=color>, but that has different constraints. Native OS X color picker has an eye dropper... that opens in sRGB mode which somewhat mitigates the issue.

... Can't interpret non-sRGB coordinates as sRGB coordinates because it will be the wrong color.

... If we return an sRGB color, it's losing information.

... But waiting for the device-independent colors to ship everywhere would block them for an indeterminate amount of time.

... Shipping in Safari, but not yet implemented in Gecko/Blink.

... Could return LAB or LCH... Display P3 would solve the immediate problem but may run into issues further down the line.

... Ideally want them to return LAB or LCH.

Alice: Makes me wonder what their stakeholders are planning to do with the picked color.

... If they clip the color at the edge of the sRGB gamut, the color will be wrong.

Lea: Difference could be up to 7 delta-E units, and 2 units is perceptible.

... P3 color space is 50% larger than sRGB by volume, and as technology evolves this difference will only increase.

Alice: there was some talk about implementing LAB/LCH in chrome, but it was blocked on plumbing the higher-definition colours throughout

Lea: WebKit has managed to keep some colors as 8-bit and some as 16bit, with a flag explaning which is which...

... We can't be unable to represent so many of the on-screen colors in CSS. People are already asking why the most saturated red in CSS isn't as bright as the brightest red on their scheme

... People using images with embedded colour profiles to display the brightest green.

... Can we block their API until LAB colors are implemented?

Alice: is there a Color object as part of the the CSS OM?

Lea: Not yet... there's a lot of discussion going around but nothing yet...

... There is need for an actual Color object.

Alice: Not clear that CSSColorValue is really representing the same thing as a potential generic Color object.

Lea: Problem is representing the user selection in a way that can round-trip.

Alice: Even just as a programmer, having the color be a string makes me sad.

... Not clear to me that the rest of the application is going to be viable without high definition colors on the web - say you're creating a web-based image editing application and you want to be able to apply colors...

... Seems like there's no "happy" solution here - either you design a special purpose extensible "color" API for this one API (or this and color pickers), OR you limit both to sRGB, OR you sit back and wait for the CSS color work to be completed.

Lea: 4th option: spec it to return an LAB string rather than an sRGB hex string. But that's still a string, so it still has that problem.

... The rest of the API, open() is asynchronous. They note that it could show a permission.

... Seems to me that this is analogous to the file picker API.

Alice: Agreed.

Lea: Also a lot of complexity about allowing you to select multiple colors. Usually when you have some UI that does that, you can exclude the UI from being picked.

... Would be much easier if it was promise-based, and then you could re-open() the picker to select another color.

Lea: Lots of implementation details around coordinates, when it's not clear what they're for, and they're not always provided depending on where the point is (whether it's in a document from a similar origin or not).

... Using an event-based model makes the API harder to use, so it should be justified with concrete use cases.

... The simple case (picking a single colour) is unnecessarily complex.

Alice: Why does multiple selection require excluding the picker?

Lea: It doesn't - but you get feedback of what color you selected. Need to show the user feedback that their action has had an effect.

... Designing the API around single colour selection would avoid errors like forgetting to close the picker or having error handling mean that the picker is left hanging.

Alice: The UI could allow the user to pick colors until they find one they like, but only return one to the application.

Points:

  • 4 ways to deal with color output, none ideal (1. wait 2. sRGB hex 3. lab() string 4. Special-purpose color API e.g. srgb property, hex property). Make sure to mention roundtripping.
  • What are the coordinates needed for? The API is designed around these coordinates (PointerEvents) but coordinates are not always provided. It's unclear what use cases the coordinates are addressing.
  • The API is designed around multiple color selection. The multiple color selection is primarily for fine tuning, but the UA can take care of fine tuning by showing appropriate UI and communicating the final color selection back to the app, which is also more privacy preserving. This would allow a promise based design and prevent bugs where the eyedropper mode is left open because something (e.g. an exception) intercepted the call to .close().

Draft comment:

We discussed this today in our VF2F and have the following feedback.

a) The biggest issue is color output. There are four ways to address this, none ideal:

  1. The Web Platform is currently in a transitional state wrt color. Possibly the best solution is to wait until there are more implementations of non gamut restricted color formats and/or a Color object.
  2. You could clip or gamut map the selected color to sRGB and return an sRGB hex value (or other sRGB color format). The problem is that color selection will not roundtrip for at least 1/3 of onscreen colors in today's displays (and as display technology improves, this will increase). Given that many eyedropper use cases need precision, we do not think this is a good solution. We also think returning strings is suboptimal. For example, a user could have a Photoshop window side-by-side with a web application and use the eyedropper to select a color from the mockup to apply to their document in the web application. The color should be identical to serve this use case.
  3. You could return an lab() or lch() string, which can specify any visible color. This does roundtrip, though is also a string. Another problem is that implementations of these formats are currently limited.
  4. You could build a special-purpose color API, with srgb and hex properties. We do not think this is ideal, as it duplicates the Color API that the Web Platform will eventually get, and it's still unclear what the primary returned value should be.

b) The API is designed around using a PointerEvent but it is unclear what the use cases are for coordinates, espcially since their presence cannot be guaranteed and selection cannot be constrained to only areas where coordinates can be returned.

c) The API is designed around multiple color selection as the primary use case, requiring use of events and explicit close(). However, as pointed out by the authors, the multiple color selection is primarily intended for fine tuning, but the UA can take care of the fine tuning by showing appropriate UI and only communicating the final color selection back to the app, which is also more privacy preserving. This would allow a promise based design and prevent bugs where the eyedropper mode is left open because something (e.g. an exception) intercepted the call to .close().

2021-03-22

Minutes

Still pending feedback

2021-03-29

Minutes

Dan: they had a discussion in breakout B.

Sangwhan: Lea left a note to pass it on to in particular Ken to take a look at the comments. She drafted a comment in the minutes. If we are all okay she will post it. Conclusion is that we will propose them to make it so it is extensible through some sort of string based mechanism that would let you denote the colour space, that comes in when you pick a colour, so instead of #ffcc00 you'd get rgb.. was the gist.. it has to be an object instead of just a string that comes in from the event.

Dan: is it backwards compatible

Sangwhan: I think yes, that was part of the main reqirements. If you can break back compat you don't have to care about.. you can invent a new colour object. Interop is part of the considerations.

Dan: then that feels fine.

Sangwhan: a lot of discussion about if it should be a promise.. request a colour and the user picks a colour the promise resolves vs an event based design (which it is right now). I found the event based design strange but don't have any better ideas. Lea wanted to hear what Ken thought. Minutes will be summarized into a comment.

Ken: [reading]

Sangwhan: they had a use case they wanted to solve which is why they use pointer events

Ken: I think it's fine what's in the minutes

2021-03-29

Minutes

Discussion about color conundrum (wide gamut, color object vs string). Consensus to add a property that returns a hex string as long as its name is explicit about what this is I.e. no generic color or colorString, but something as explicit as hex, hexString, srgbString and so on.

Side discussion on Color object, which WG to work on it (CSS WG, Houdini, or new?). Consensus that it should not be a new group, and that it can happen in either CSS or Houdini, this should not hold up the work.

We are still concerned about the event based design and think a promise-based design would be more suitable.

With a promise based design, open() can allow for a permissions prompt or not, based on what the UA wants to expose. E.g. if drag and scrub is allowed, it would make sense to have a permissions prompt, but if the UI only allows selecting individual colors by clicking, it can be omitted.

Peter: security issues, sites maliciously opening eyedropper to sniff can be used for nefarious reasons by advertisers

Lea: should only active tab be able to open the eyedropper?

Peter: user activation state?

Rossen: this is already in their explainer

Discussion on refining color state by multiple clicks. Should the API allow multiple color selection, or should it be up to the UA to provide UI for this? Consensus around previous feedback (API designed around single returned color, UA can provide UI for refinement)

2021-05-Arakeen

Minutes

Looks stalled, posted to ping them.

2021-08-30

Minutes

Lea: they made a bunch of changes...

Ken: ... middle of the lens... implementation ...

Lea: there's also a spec that is not linked in the issue.

Lea: thoughts: you notice there's an open method - there's no close method. so no way for a webapp to close an eyedropper it's opened. seems strange. Also - if a user exits eyedropper mode without making a selection the eyedropper rejects with an abort error dom exception... I don't think this is an error condition. Vs. resolve with null. Peter seemed to have same opinion...

Dan: +1

Ken: +1

Lea: Sangwhan also said it should not reject. The new proposal is "rejected".

Ken: I guess some developers might not consider you get a valid color back? It's an abort.

Peter: I could see abort being thrown if it's something else that aborts it

Rossen: Yes how is it not regular user flow?

Ken: we should see how it works with filesystem access....?

Ken: file picker is consistent with reject...

Dan: consistency... Maybe give them the info and ask them to think about it?

Ken: as a developer I like it.

Peter: let's not set a precedent with one example....

Ken: I think people get confused about it... Looking back it should not be called error...

Peter: it's a question - are you using exceptions to signal error...

Ken: is the rejection of a promise an exception?

Peter: yes supposed to be.

Lea: yes.

Ken: if i want a color and don't get a color isn't that an exception?

Pete: should "cancel" file an exception?

Rossen: nothing exceptional about it - it's in normal user flow.

Lea: also handling exceptions is more heavyweight.

[vigorous debate ensues]

Peter: we should be consistent.... exception handling more heavyweight in JS

Rossen: in our principles doc

Dan: https://www.w3.org/2001/tag/doc/promises-guide#api-design-guidance

Peter: https://github.com/w3ctag/design-principles/issues/55

Lea: one of their use cases was a drawing application -- they had an event based design... we gave feedback this is not an appropriate design .. now they have a promise based design but this use case can't be preserve... on the other hand these types of app could implement their own eyedropper... Also not sure what to suggest...

Dan: might be something we should note back to them....

Peter: passing an abort controller into the open method rather than giving it a close method might be a better approach.

Glad to see the improvements in both API shape and spec text!

We discussed this again in our plenary today and while we thought this was an improvement over the previous iteration, we did have a few thoughts and questions.

While it's good that there is some guidance for UAs on when to exit eyedropper mode without a selection (e.g. pressing ESC), it might be good to also provide a way for the application to explicitly close the eyedropper, e.g. a `close()` method or an `AbortSignal` parameter. Since no UI events are dispatched this would not be useful for closing the eyedropper through user interaction (e.g. via additional keyboard shortcuts), but it could be useful for closing the eyedropper (with no selection) in case another high-priority event occurs in the application, e.g. a modal dialog.

Some of us were surprised by the [promise rejection  when no selection is made](https://wicg.github.io/eyedropper-api/#eyedropper-interface) (vs resolving with `null`), since this is [ordinary use and not an unusual bad state](https://www.w3.org/2001/tag/doc/promises-guide#rejections-should-be-exceptional). Do you consider exiting without a color selection an error condition? Did you choose this design to be consistent with another API?

While we think the promise-based design is an improvement, we did notice that it does not address your original use case of picking colors from a drawing application. Only the final composited color will be picked with no means of tracing it back to the top-most color that the user may have intended to pick since no UI events are fired throughout the selection. What are your thoughts on this? Is there another way to accomplish this with the new design or did you decide that this is a non-goal for now?
Also note that for multiple color selection to be possible with a promise-based API like this, subsequent (synchronous) `open()` calls would need to open the eyedropper in the same place it was before. I don't see anything about this in the explainer or the spec.

Btw while `sRGBHex` definitely addresses our feedback on future-proofing this API, do note that a simpler `hex` property would also do, since all hex colors are in sRGB anyway. That also avoids the casing debates raised above.

Relevant, note that there is already an effort to standardize a Color API for the Web Platform by a subset of the CSS WG: https://github.com/WICG/color-api and serving this API is listed as an explicit goal.

2021-09-Gethen

Minutes

[we closed it]

Sangwhan: we don't have a resolution for return errors vs throwing. Right now no concrete guidance on this. This is related to issue 55 in Design Principles..