#433: WebCodecs

Visit on Github.

Opened Oct 18, 2019

Hello TAG!

I'm requesting a TAG review of:

Further details:

You should also know that...

  • The review is early on in the design stage because we expect to learn a lot from early implementation feedback, so we'd like to get a rough design reviewed so that we can begin implementations and get that feedback to further refine the design.
  • A portion of the explainer and WebIDL could be done a separate standard from WebCodecs, namely the conversions from WHATWG streams to MediaStreamTracks (called TrackWriters and TrackReaders at the moment). Many use cases require these, but use cases outside of WebCodecs could also find these useful. For simplicity, we are considering such work inside the scope of WebCodecs, although it could be pulled out later.

We'd prefer the TAG provide feedback as (please select one):

  • open issues in our GitHub repo for each point of feedback

Discussions

Discussed Nov 19, 2019 (See Github)

Dan: David?

David: Looked briefly - it seems probably good for a high level perspective but I think we should maybe look closer if we have the expertise.

Dan: If we do this at the F2F we should get an external expert.

Comment by @dbaron Jan 11, 2020 (See Github)

I've read through the explainer (twice, I think) and I don't have anything to complain about so far.

Comment by @cynthia Mar 5, 2020 (See Github)

I've also looked at this and think it would be a useful addition to the platform. One question I do have is if the first deliverable could also support images - there is [1] HTMLImageElement.decode() but that really doesn't do exactly what one would expect. Could any of the editors comment on if this is possible?

CC: @steveanton, @padenot, @aboba, @pthatcherg

[1] https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/decode

Comment by @padenot Mar 5, 2020 (See Github)

I'm pretty sure this would be possible. More and more, image codecs use video codec technology/bitstream.

Discussed Mar 16, 2020 (See Github)

Scribe: Alice (with assistance)

Tess: I took a look at this earlier... not actually assigned to me but I took a look anyway. Maybe it could be a propose close? Sangwhan had one question, it's been answered, and David thought it looked fine.

... I started looking into it a little bit... this is in the Intent to Implement:

Safari: Negative signals (concerns over keeping it fingerprinting neutral)

... Should we take another look from the perspective of fingerprinting?

Sangwhan: I don't understand how this is another fingerprint. Either you'd be have (less than a handful of) silicon provided encoders or you'd have ffmpeg. The codec availability at most gives you as much entropy as knowing say, your phone model number which we happily dump on the UA string

Tess: I'm going to dig up the minutes from TPAC to find where the fingerprinting conversation happened, and what the specific concerns were, so we can follow up in our issue

Discussed Mar 23, 2020 (See Github)

[bumped but we can still chat in the plenary

Discussed Apr 20, 2020 (See Github)

Discussed closing at the plenary.

Peter raised a question about using these codec apis in existing codec usage, such as MediaRecorder.

Comment by @plinss Apr 20, 2020 (See Github)

Given that these APIs offer more control than existing places where codecs are used, such as MediaRecorder, has any thought been given to exposing the existing codes via these APIs? For example, I can see starting a MediaRecorder, getting the video encoder and setting options (or swapping an encoder I construct), rather then extending the MediaRecorder APIs to have similar control surface...

Comment by @cynthia Apr 22, 2020 (See Github)

Raised https://github.com/WICG/web-codecs/issues/50 to track the image use case.

Comment by @dbaron May 26, 2020 (See Github)

@cynthia and I are looking at this in a breakout at the TAG's virtual face-to-face.

We (the TAG) have looked at this a few times, and I think we're generally pretty happy with it. The design looks appealingly simple -- which is either great (if it can stay simple in the end), or it's a sign that a bunch of things were missed in the design. We're hoping it's the former.

Peter's comment gets at one of the tricky issues with adding new fundamental APIs that other things are built on, when those other things already exist. When that happens, there's generally a process of making the existing features, like MediaRecorder, work with the new fundamental API. That doesn't need to happen all at once, but it does often require somebody making sure that progress happens (e.g. by filing appropriate issues on places that will need to be changed, so we don't lose track) so that the desired end state can be reached. We hope that happens here once the foundation is ready.

So I think we're going to close this issue. Thanks requesting our feedback. Let us know if there's anything else you'd like us to look at later on in the process.