#240: The web platform needs a service discovery mechanism

Visit on Github.

Opened Apr 7, 2018

While reviewing #237 in the Tokyo F2F, it came to our attention that the platform lacks a good mechanism for service discovery.

One of the ideas that came out of the discussion was to look into what the .well-known "bootstrapping" is attempting to solve, and prototype some of the ideas we had on top of DNS SRV records. (and possibly publish as a finding?)

Discussions

Comment by @mamund Apr 11, 2018 (See Github)

Over the last month or so I've been working on a PoC on "open discovery" for the Web: http://www.open-disco.org/

Not sure if this is along the lines you were thinking and would be happy to share my code and other notes with anyone who might be interested.

Comment by @plinss Jul 26, 2018 (See Github)

@plinss to write up a blog post about service discovery, submit to w3c strategy funnel

Comment by @torgo Feb 5, 2019 (See Github)

@cynthia to confab with @plinss

Comment by @hadleybeeman May 22, 2019 (See Github)

Discussed at Iceland f2f.

@cynthia is going to arrange a breakout at TPAC with @plinss and interested parties, to flesh out what devices need to find each other. Probably with the media use case first; finding server; finding set top/desktop.

(@chrisn? @anssiko? Are you interested? Anyone else?)

Comment by @kenchris May 22, 2019 (See Github)

@larsgk might be interested

Comment by @mamund May 22, 2019 (See Github)

I'd be interested in this. been working on a service-level discovery pattern for open web here: http://www.open-disco.org/.

Comment by @anssiko May 22, 2019 (See Github)

Second Screen CG has been working on the Open Screen Protocol https://webscreens.github.io/openscreenprotocol/ for devices that can render web content like Internet-connected TVs, HDMI dongles. OSP-compliant devices use DNS-SD over mDNS to find each other, QUIC for transport and metadata discovery. The spec is now considered ”feature complete” CG draft and is being prepared for TAG review. We have F2F https://www.w3.org/wiki/Second_Screen/Meetings/May_2019_F2F tomorrow, so I’ll put this issue on the F2F agenda.

@mfoltzgoogle would be a key contributor to this proposed TPAC breakout session.

Comment by @chrisn May 23, 2019 (See Github)

Thanks @hadleybeeman, I'm certainly interested.

Comment by @larsgk May 23, 2019 (See Github)

Thanks @kenchris - VERY relevant, thanks!

Comment by @cynthia May 23, 2019 (See Github)

Just to be clear, the part that this touches on is the hardcoded part in section 3 in the OSP spec. I’m not particularly thrilled about exposing mDNS (with all it’s quirks) over the web platform, but it is indeed a viable option since it has extremely wide adoption. (We previously attempted this in the past with NFD spec, but I don’t consider that a success.) What we would like to see come out of the discussion is the following:

  1. A common API interface/pattern for designing service discover-like features (e.g. local network and bluetooth might be examples where we can have a similar API pattern), sort of like what we did with streams
  2. A mechanism for exposing/discovering network services (e.g. query default dns resolver, mDNS, etc.)

Do note that this is not a place where we are gathering proposals, it’s more of a architectural concern issue where we want to not do N inconsistent API patterns, and end up with a pile of regret later.

We mentioned media because it feels like the most immediately use case which can benefit from this - as a immediate example, Plex does a lot of interesting (en_UK interesting) things to work around this not being available; so there is a clear need.

Should we setup a separate call to discuss if this is worth a plenary breakout during TPAC later this year?

Comment by @ylafon May 28, 2020 (See Github)

On top of this, discovery mechanism should ideally have measures to avoid doing "service scanning", like by poking too much in well-known URIs, something equivalent to port scanning.

Discussed Jan 1, 2021 (See Github)

Hadley: This was waiting for a blog post by Peter and TPAC breakouts in 2019 with Peter and Sangwhan. We should ask them in plenary where this is and what to do about it.

Discussed Sep 1, 2021 (See Github)

Peter: has anything happened?

Dan: comments recently..

Peter: ... MDNS is part of it - but the other half is SRV records - DNS records - there adre a number of federation protocols - e.g. matrix - you do a SRV lookup on xyz.com and find out there is a matrix server here... because that doesn't work on the web everyone has to use a well known URI. We don't want to give broad access to DNS to a web client - but limited access to e.g. SRV lookups would be useful.

Dan: when I type in w3.org there's already a record which points my browser to the appropriate ip address to make the http request.. why do we need something .. a second level of indirection where we?

PeteR: that's an A record.. for matrix, I run a matrix server, but my matrix id is peter:linss.com .. how do you find my matrix server? the standard way to do that is look up the service record

Dan: but that's a record for a specific entity

Peter: it's a record for the service

Dan: you at that service.. you described a string that is you at that service

Peter: the record name is _matrix._tcp ... it's a pattern for service records. Anybody gives you an address that ends inlinss.com it looks that service record

Dan: why is that useful for the web?

PeteR: it's about you typing in my matrix id..

Amy: something about acitivitypub

Dan: I'm talking about how people use the web in the browser

Peter: if you want to build a web client you need to be able to do this level of discovery. If i want to put something on your calendar you need to be able to type my email address into your calendar

Amy: in the social web we use relations - you have the profile URL - your name at your domain.. you look that up and get a link relation in the header or the HTML -

Peter: you have to build a http level protocol so the clients can use this. Matrix did the same - but it's a pain to put a well known URL in linss.com... same person might not own the matrix server...

Amy: that's why we didn't use well known urls in Social Web WG ...

Peter: single scoped DNS lookup... Don't want to add general DNS capabilities to the browser. Doesn't change how I would pull up a web site. If I type w3.org it could require a SRV look up but i don't propose that. it's for service - e.g. a calendar.

Dan: right now if I want to subscribe to a calendar I need a full URL. It might simplify thigns like subscribing to a calendar, which in some cases is not using a traditional web browser but some other client. In some cases you can take a calendar url and subscribe from a calendar app or inside a web app. Either way you've got a full url.

Peter: I can give you a url to my calendar.. you should be able to type in my email address and a web client should be able to figure that out by doing service discovery, because my servers do advertise the correct dns and txt records to do that lookup. A native calendar app can do that, a web calendar app cannot

Dan: in the world where people who create calendars and advertise those calendar urls, where they're all calendar.google.com how does that help me?

Peter: it deson't help to have your calendar on google.com....

Dan: I know that. But if I change my calendar from one cloud service to another, it's still hosted by somebody but it's at my torgo.com address..

Peter: you update the service record at torgo.com and I can go find your calendar server, whichever one it is

Dan: how does it help people who use highly used services on the web right now? vs people who want to tinker around with their own xmpp servers

Peter: it gives you a level of indirection so I don't have to advertise the fact I'm using googles calendar, just the fact that I have a calendar. And that gives me freedom to move my calendar around without breaking all the links. I'm asking for two method calls on the browser, one to do an mdns query looking for a local service and one to do a service record lookup. We don't have to call it that, but I sholud be able to pass in a domain and the name of a service and under the hood it does a srv lookup and a txt record lookup and it gives me the url, port, domain. That's all this is about. Clients can't do that now cos they can't do dns lookups.

Dan: is there a browser that is close to being able to do this? what does firefox do?

Peter: nobody I know of has anything like this. If you know of a DOH server you can implement this clientside. You can do any dns query over http but that's it's own nightmare.

Reviewing the Chromium proposal on this https://developer.chrome.com/docs/extensions/reference/mdns/ which is marked as deprecated

Peter: doesn't do SRV records...

Dan: i wonder if it was deprecated because of security concerns

Peter: this is entirely about local discovery

Dan: we talked about local network accses but even thatn eeds to be a picker UI so you're not allowing random web apps to scan your local network. I don't think there's a similar issue with what you're taling about. Just giving the client ability to do dns queries

Peter: public information..

Peter: sangwhan and I discussed just writing a polyfill and seeing if anybody cares at wicg

Dan: I wonder if Andrew Beck would be interested in helping with that, coming at things from the cdn perspective. Keep it open as a placeholder for that activity?

Peter: sort of. I don't think it's the TAG's job to design and build this feature. But something I could do on my own because I care.. as far as the TAG's role, we've identified a hole in the platform. I thin it's our job to do that and call it out and try to get somebody in w3c to work on it.

Dan: one mechanism could be a finding... or a blog post. More public than an issue register

Peter: maybe a blogpost. Also w3c strategy funnel..

Dan: action is on Peter to write a blog post

Peter: k

Comment by @torgo Sep 7, 2021 (See Github)

Discussed today possible outcomes of this issue since the TAG likely won't itself take up work in this area. We discussed raising a process issue, raising it to the team, other options...

Comment by @cynthia Sep 7, 2021 (See Github)

Let me try to find out what happened with this https://developer.chrome.com/docs/extensions/reference/mdns/ - might be a start.

(on non-TAG capacity)

Comment by @hadleybeeman Feb 8, 2023 (See Github)

We discussed this at our W3CTAG Moon base alpha face-to-face.

We agreed that this work should be incubated in a community group. @plinss has taken on that task. I will set this this to proposed closing, and we intend to close it when the CG is spun up and the discussion can move there.

Comment by @plinss Feb 8, 2023 (See Github)

CG proposed looking for supporters...

Comment by @plinss Apr 18, 2023 (See Github)

CG created