#358: MediaStream Image Capture API PTZ (Pan/Tilt/Zoom feature)

Visit on Github.

Opened Mar 26, 2019

Góðan dag TAG！

I'm requesting a TAG review of:

Code Snippet:

if (!('pan' in capabilities)) {
  return Promise.reject('pan is not supported by ' + track.label);
}
inputPanRange.oninput = function(event) {
  track.applyConstraints({
    advanced: [{
      pan: event.target.value
    }]
  });
}

Explainer/Motivation: Many cameras have the ability to Pan and Tilt which is specially useful in the video conferencing(WebRTC), like steering the camera to face the speaker, etc. FWIW Chrome has been using a private extension(webcam_private.idl) to satisfy this use case.

Further details (optional):

You should also know that...

Image Capture API is already shipping in Chrome 59 on Android and desktop. Pan/Tilt feature is a small addition to the property set for capabilities, constraints and settings.

We'd prefer the TAG provide feedback as (please select one):

  • open issues in our Github repo for each point of feedback
  • open a single issue in our Github repo for the entire review
  • leave review feedback as a comment in this issue and @-notify [github usernames]

Discussions

2019-04-17

Minutes

Torgo: Kenneth, the thing you brought up... The Tilt Feature? Want to explain this to us?

Sangwhan: This isn't high priority for us, except that Blink intent to ship is delayed for our trip. But they do intend to implement and ship at the same time which is unusual.

...This is an extra capabiltity for webcam control -- rotate it, point to an object you want to focus on. Thus far, it's been proprietary APIs in Chrome.

...This is the first time we've introduced something something that physically moves into the web access

Torgo: moves and sees

Sangwhan: it does require you to grant access to the camera though. Which I think is good.

...They use slightly strange units, for reason. Arc-seconds, which is what UBC uses. But on Windows, half APIs are degrees based and half are seconds-based.

...Do we have plans to add more things that rotate into the platform? And are they okay with arc-seconds?

David: CSS has multiple angle units (e.g., for transform, radial gradients) and none of them are arc-seconds.

Torgo: Where is this? a PR on media capture API?

Sangwhan: Yeah.

Torgo: but this has been merged in the actual draft version?

Sangwhan: I believe so

Torgo: I just want to make sure this isn't only in a googe doc somewhere that we don't have access to

Sangwhan: No, it's been added to media capture.

Torgo: okay. Did you get a response about units?

Sangwhan: I understand wh ythey're doing this. it's because of rounding error potentials if you're using larger units. I think it's fine, but I don't think anything else on the web platform uses it.

...But given that the other things that also rotate are servo motors, I thin kthis is a good compromise.

DAvid: This ought to be designed so the capabilites of hte hw can change. today, the cameras in the laptop basically point straight ahead. But in the future they may rotate... and people will want permission to rotate it.

Torgo: I just saw a device with a camera that has the same housing, with a camera that flips from looking at you and looking away. Could be a privacy issue, "I granted you permission to see me, but not the room that I'm in". but if the remote user can tell the camera to move around... normally that's under user control. That seems like it would go against user expectation.

sangwhan: re david's question re extensibility... the constraint is an object.

Re privacy concerns... yeah

David: More... could permission prompts be added without breaking it. In other words, if it has to be synchronous without a prompt it nthe middle...I'll comment in the issue

Torgo: i'll add my privacy concern there

sangwhan: applied constraints does the rotation. and it is asych and uses a promise. Could have a permission, but isn't in the spec now.

Torgo: feels to me like it should be possible for implementations to add a permission where it fits the use case

sangwhan: it could look at hte constraints being passed over and say "these are dangerous". But that's later; it's not possible now.

...Add me to it.

This is being implemented by Intel.

Torgo: I'll put you and Kenneth on it. Let's come back to it on the 24th

Peter: this is only part of hte image capture? So if you have a video stream and you're just trying to get control of tilt for an image capture to take a high res photo in the middle of a stream... what does it mean?

sangwhan: you'll have to set it twice.

Peter: so there's no way of moving the camera as part of a regular RTC stream without taking a photo?

sanghwan: you'd have to set up the RTC stream first

Peter: right, but can I move the camera without grabbing the image?

sanghwan it's on the video track, actually. that correcsponds to the device, and you add the constraint to the track.

(Assigning the rest of the new issues to TAG members

2019-04-24

Minutes

Kenneth: additional permission prompts?

Sangwhan: don't think that's a good user experience.

Peter: happy-ish with the feedback to my comment.

Sangwhan: from an implementation perspective - if you want to have it separately then - underlying native api had to hold on to same object...

Peter: David did they address the comment about the prompt?

David: I haven't dug into the spec to know what that means.

Dan: fold two prompts into one?

Sangwhan: how?

Peter: you get one prompt when you invoke getusermedia. Theoretically you could get one prompt that asks for video feed and pan/tilt... then habe to re-apply constraints.

Kenneth: you will see what they are looking at.

Dan: yes if there is self-view.

Hadley: it feels weird to me - violating expectations and confusing users.

Sangwhan: I haven't seen other examples. Personally I think it's something that could be patched in later.

Kenneth: i think it's ok to have a notifciation "remote user is trying to move your camera"

Kenneth: professional google hangouts does that, right. Some will use machine learning to focus on who is speaking, for example...

Sangwhan: there are restrictions from blocking other party from messing with your camera.

Dan: blah blah blah ethical web

Sangwhan: permissions in the web are pretty bad. If we had a non-broken permissions. I'm OK with this shipping then we can add a permission later.

Dan: i think we should be challenging them to come back with a solution that addresses these concerns..

Hadley: i second.

Dan: this

Hadley: i'm conscious of how often you joke about when you take video calls what is off camera. I worry about violating that at a platform level - that doesn't fit with the user being in control of their experience of the web.

Peter: i agree with that but there are valid uses for remtoe control. I can imagine a camera that flips back and forth between front and rear facing - but that'd be a different API.

Kenenth: on new phones you have "selfie lens"... like wider angle view.

Peter: i beliebe there's a zoom feature here already

Sangwhan: there is.

Dan: isn't it desirable that we should be asking the developer to ask multiple permissions?

David: the UA might be able to bundle or could split them out but only if the API is

Hadley: I'm comfortable leaving that to the user agent.

David: In order to leave the choice of whether to bundle the permissions to the user agent, the user agent would need all the information at the time it's asking the user for the first permission.

Sangwhan: that makes sense. The feedback we would liek to bring back is that we'd like it o be a secondary permission but so you can ask for that permission in a bundled way.

David: assuming there's a reasonable way to do it.

Peter: when you call getusermedia you pass in a set of constraints that you're asking for - you try to set pan/tilt constraints then you could bundle it in one prompt.

David: that's even the sort of thing that could be patched later but it might be good to get it up front

2019-05-01

Minutes

Pending external feedback

2019-05-08

Minutes

Peter: Marked pending feedback... We got a "thanks" - and Ken and Sangwhan aren't here. I'll bump

2019-05-15

Minutes

Nobody assigned is present.

2019-06-19

Minutes

Sangwhan: Waiting on feedback on this one.

Dan: It says review complete...

Sangwhan: we reviewed everything they brought us...

... milestone empty for now, I'll add it to an agenda when they get back to us.

2020-03-16

Minutes

Alice: No explainer - but came in before that requirement

Ken: Pretty sure nothing has happened since (last comment). This is a powerful feature that can be abused.. It would be nice if they wrote an explainer... Security & privacy questionnaire responses also not correct... it says "no" to everything...

Dan: on s&p, 3.2 (data) and you can use this to gain access to data that the user has not intended you to see... 3.4 (cross-origin-state) they have said no, but can you read the angle?

Ken: looks like you can read the angle..

2020-04-06

Minutes

No explainer still; milestone changed to next wee

2020-04-13

Minutes

Ken: they haven't done the explainer yet. I pinged them again.

Sangwhan: if they don't come back by our next milestone, suggest we close

2020-04-27

Minutes

Ken: i have progressed this.. We have an explainer now... it's pretty OK. You will get a dialog... up to implementation whether nor not if you deny permission the app will still get regular camera access.

... you will get video permission... then you (app dev) will have to check if you have access to pan/tilt...

Dan: that's to mitigate the fingerprinting...?

Ken: yes but also not to force users to accept a powerful permission. It's a specific other set of permissions that you will be shown in the permission prompt. Needs to work if you do or don't have the camera plugged in... Seems to be fulfilled.

... other thing... when a web site is not in focus (when you are not looking at the screen) then these capabilities will be disabled. Not yet in the explainer but they said they would consider it.

Dan: how much of what you just said is in the spec vs the implementation?

Ken: Not sure. They [chromium] are doing the implementation now.

Dan: [left a comment and bumped to next call]

2020-05-11

Minutes

Dan: [pings the issue to see if there have been any progress...]