HIQaH! QaH! TAG!

I'm requesting a TAG review of MediaStreamTrack Insertable Media Processing using Streams.

We propose an API that allows application-level code to access the raw (unencoded) video or audio data for a MediaStreamTrack object. The idea is to allow Web applications to implement custom sources and sinks, following the source-track-sink paradigm on which the MediaStream spec is based. This will enable applications to 1) connect existing tracks to custom source (e.g., a camera track to a custom codec), 2) create new types of tracks that can be sent to existing sinks (e.g., peer connections), and 3) combine both mechanisms to perform custom processing on existing tracks (e.g., take an existing track, process its raw video frames to add an effect, and create a new track using the processed frames). This API leverages the Stream API and interfaces for raw media data proposed by the WebCodecs API, and is a logical follow-up to WebRTC Insertable Streams, which provides access to encoded data

• Explainer¹ (minimally containing user needs and example code): https://github.com/w3c/mediacapture-insertable-streams/blob/main/explainer.md
• Specification URL: https://w3c.github.io/mediacapture-insertable-streams/
• Tests: WPT not yet available, coming soon
• Security and Privacy self-review²: https://github.com/w3c/mediacapture-insertable-streams/blob/main/tag_sp.md
• GitHub repo (if you prefer feedback filed there): https://w3c.github.io/mediacapture-insertable-streams/ (we don't require feedback there)
• Primary contacts (and their relationship to the specification):
  • Guido Urdaneta, (@guidou), Google, Chromium implementer, editor (guidou@chromium.org)
  • Harald Alvestrand, (@alvestrand), Google, Chromium implementer, editor (guidou@chromium.org)
• Organization(s)/project(s) driving the specification: Chromium & W3C WebRTC WG
• Key pieces of existing multi-stakeholder review or discussion of this specification: WebRTC WG Minutes
• External status/issue trackers for this specification (publicly visible, e.g. Chrome Status): Chrome Status, GitHub

Further details:

• I have reviewed the TAG's API Design Principles
• Relevant time constraints or deadlines: It would be nice to have a reply by April 8, 2021.
• The group where the work on this specification is currently being done: W3C WebRTC Working Group
• The group where standardization of this work is intended to be done (if current group is a community group or other incubation venue): W3C WebRTC Working Group
• Major unresolved issues with or opposition to this specification: There is an open issue about using streams vs. giving stream-like properties to MediaStreamTracks.
• This work is being funded by: Google

We'd prefer the TAG provide feedback as: 💬 leave review feedback as a comment in this issue and @-notify @guidou and @alvestrand