Design Review #495

#495: AVIF Decode

Visit on Github.

Opened Apr 7, 2020

Hello TAG!

I'm requesting a TAG review of AVIF Decode.

Specification URL: https://aomediacodec.github.io/av1-avif/
Primary contacts:
- AOMedia for the specification
- @wantehchang (Google) @jaikrishnan (Google) for the implementation
Organization(s)/project(s) driving the specification: AOMedia
Key pieces of existing multi-stakeholder review or discussion of this specification: https://github.com/AOMediaCodec/av1-avif/issues
External status/issue trackers for this specification (publicly visible, e.g. Chrome Status):
- crbug
- bugzilla

Further details:

I have reviewed the TAG's API Design Principles
Relevant time constraints or deadlines: N/A
The group where the work on this specification is currently being done:
The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue): Alliance for Open Media
Major unresolved issues with or opposition to this specification: N/A
This work is being funded by: N/A

You should also know that...

N/A We'd prefer the TAG provide feedback as (please delete all but the desired option):

💬 leave review feedback as a comment in this issue and @-notify [github usernames]

Discussions

Discussed Apr 1, 2020 (See Github)

Dan: AVIF decoder -

Ken: I looked through this... left some comments.. on what else we need to change

Peter: seems we're lacking the proper layering...

Tess: that's what the web codecs API is supposed to do...

David: this is just a new image format...

David: web codecs is more centered around audio & video...

[discussion of how this relates to web codecs]

Peter: I would like to see - before we add new black boxes - let's make these things extensible. E.g. with webaudio...

David: other question is - what APIs do you need?

Ken: hardware acceleration - which you don't get with webassembl

Discussed Apr 1, 2020 (See Github)

Sangwhan: what do they want? I'm looking at the chrome issue tracker - this is transparent decoding.

Yves: this is in line with the crypto one - depends on the buy-in and patent related issue... not really technical.

Sangwhan: they are adding a new image format ... it's a browser feature. From a greater perspective, it should be "is the web going to add a new image format"?

Ken: too early to do a recommendation for that.

Sangwhan: this doesn't ask for any of that.

Dan: shall we propose close, summarily, and just say "it's not in TAG's wheelhouse"?

Ken: better format than web-p...

Dan: is this related to HEIC?

Sangwhan: No. Related to AV1. Key frame format for AV1.

Ken: supported by Netflix - much better quality than JPEG

Sangwhan: no hardware support. WebP has some level of hardware decoding.

Sangwhan: as a recommended format, I think we need to consider it, but as a pure technical perspective, let's not consider.

Ken: this is really good for the web because it's much more open....

Sangwhan: we should look at this in the sense of : is this an official format for the web? The web platform does definitely need a better image format.

Dan: other dependencies?

Sangwhan: clipboard

Ken: canvas API

Sangwhan: another API: Web codecs

Ken: also multiple file extensions... why? That's gonna be a pain.

Discussed Apr 1, 2020 (See Github)

Sangwhan: i don't know what they want reviewed

Ken: AOM - AV1 - for image format - these are from mpeg... heif - this is related to patents

Sangwhan: policy wise we do not discuss patents

Dan: Agree

Yves: from what I've heard - avif is believed to be RF. But not proven. And HEIF has patents.

Ken: if you read the spec, the container is based on heif .. what is inside the container is av1 based. this is based on isobmff ...

Ken: how should we ask that question... will write it.

[bumped]

Discussed Apr 1, 2020 (See Github)

Ken: No update yet.

Yves: mark as pending external feedback...

Ken: if we're not getting answers ... we should ping them...

Yves: Chris Lilley is looking into it.

[bumped]

Comment by @cynthia Apr 14, 2020 (See Github)

Hi @jaikrishnan!

It's pretty unclear what we are expected to do here; there is no explainer, nor from our side an API to review. From what we see it's the browser side implementation to allow decoding a new image format - we don't quite have a review process for these cases.

This does introduce a new image format onto the web platform, which means that any form of I/O that allows a still image (e.g. canvas, clipboard) would probably require some level plumbing to support it (e.g. registries, whitelists) once it becomes something beyond a blackbox codec that is only expected to be used as decoded internal state of an HTMLImageElement. (e.g. SkImage?)

Comment by @kenchris Apr 14, 2020 (See Github)

This seems somewhat related https://www.w3.org/TR/mse-byte-stream-format-isobmff/

Also maybe it would be nice explaining where these other standards used are used today. It seems to be used for MP4 files today. If such info and other background could be part of the intro in the spec that would be nice, if not, it definitely belongs as part of the explainer document.

In cases like "in the generic image file format [HEIF]" it would be nice to spell out what HEIF actually stands for like High Efficient Image Format - that might not be clear to the reader.

Does containers like ISO-BMFF and HEIF follow same patent policies etc as AV1 video format?

From a web perspective, I think it would be nice to know if there are any parts of the platform that needs to be patches to support AVIF property. Like you can convert an offscreen canvas to say a PNG and set certain settings while doing so: https://developer.mozilla.org/en-US/docs/Web/API/OffscreenCanvas/OffscreenCanvas.convertToBlob()

Those settings might be different for AVIF and I think that should all be considered while proposing adding AVIF to the web platform.

Comment by @kenchris Apr 20, 2020 (See Github)

Does AVIF support alpha channel?

Comment by @wantehchang Apr 21, 2020 (See Github)

On Mon, Apr 20, 2020 at 8:33 AM Kenneth Rohde Christiansen notifications@github.com wrote:

Does AVIF support alpha channel?

Yes, it does.

Wan-Teh

Comment by @kenchris Apr 21, 2020 (See Github)

Could you explain why the HEIF container was used in comparison with other containers (WebP uses RIFF). How does this choice affect the use on the web?

Comment by @cynthia Apr 21, 2020 (See Github)

So a bit of a blunt question based on what we discussed in today's call - this is based off of HEIF, which is not a royalty-free standard as we understand it. What was the technical background of the decision, and is there a path forward given the provenance of the core container format this is based off of?

Comment by @cynthia Apr 21, 2020 (See Github)

(Ugh, for some reason I was seeing a cached version without @kenchris's comment.)

Comment by @kenchris Apr 28, 2020 (See Github)

@wantehchang @jaikrishnan any update to our comments/questions?

Comment by @jaikrishnan Apr 29, 2020 (See Github)

Could you explain why the HEIF container was used in comparison with other containers (WebP uses RIFF). How does this choice affect the use on the web?

Benefits of the container design are primarily familiarity for developers, particularly those whose pipelines include uploads from mobile cameras.

So a bit of a blunt question based on what we discussed in today's call - this is based off of AVIF, which is not a royalty-free standard as we understand it. What was the technical background of the decision, and is there a path forward given the provenance of the core container format this is based off of?

Re: provenance, the AVIF image file format describes the use of an AV1 bitstream with a subset of the ISO-BMFF container, i.e. the MiAF constraints. Our view is the AVIF format would be available for use by open platforms like Chromium, in line with AOM's position on AV1 royalties.

Comment by @jaikrishnan Apr 29, 2020 (See Github)

It's pretty unclear what we are expected to do here; there is no explainer, nor from our side an API to review. From what we see it's the browser side implementation to allow decoding a new image format - we don't quite have a review process for these cases.

This is a good point, let me confirm with blink-owners if AVIF should go through a TAG Review, and then add an explainer and other design context that would be helpful

Discussed May 1, 2020 (See Github)

Ken: [researches ship status in Chromium]

Sangwhan: they wanted to check with Blink owners ... was that thread updated?

Ken: no, not mentioned here...

Comment by @cynthia May 12, 2020 (See Github)

@jaikrishnan Did you hear anything back? We'd be happy to review as long as it's clear what aspect we should be reviewing.

Comment by @jaikrishnan Jun 24, 2020 (See Github)

Thanks @cynthia, we looped back with blink-api-owners and confirmed that a Tag Review wouldn't be applicable for an image decoder implementation. I'll close this issue and note this in the intent to ship.