#390: Native File System API
Discussions
Comment by @kenchris Jun 27, 2019 (See Github)
I see some new API ideas here: https://github.com/WICG/native-file-system/issues/19#issuecomment-490579093
What is the status of that, will that be changed in the spec or not? because that affects review.
Also, I find inPlace
confusing. It doesn't explain what inPlace
does, and repeats the description of keepExistingData
's behavior. Our understanding so far is that inPlace = true
means it will write directly to the destination file without creating a temporary copy first, but it's not clear. So potentially faster, but I though the copy was there because of security and integrity - I understood you never wanted to write incomplete data.
Discussed
Jul 10, 2019 (See Github)
Dan: danger will robinson
Hadley: [danger noises]
Dave: 5th time?
Hadley: would lukasz be interested in joining?
Dan: this might have privacy issues.
Lukasz: yes.
[milestone set for 3 weeks
Comment by @lknik Jul 22, 2019 (See Github)
In the security/privacy questionnaire, what do you mean by state as in in "The drive-by web will only have enough state to allow it to re-prompt for access, but the access itself won't be persistent.?
On "No, unless a device exposes such sensors as files or directories. User agents are encouraged to block access to such files or directories (for example /dev on linux like systems).", where in the specification UAs are encouraged to block the access? I think this is 5.1 but not stated directly?
I also see that in PWA mode this API will behave differently. There are not many such APIs as of now but I believe it would be useful to have the differences (permissions model) documented in the spec.
Discussed
Jul 31, 2019 (See Github)
Kenneth: Apart from contentious parts, API seems nice. Filed a few issues; haven't gotten response yet. Spec seems early, they want an origin trial soon. Spec just has API, not a whole lot of other info there.
Sangwhan: The API itself is fine -- the "just a javascript API". I'm not entirely convinced about the mitigations that are in place to prevent terrible things happening to your home directory. The getDirectory
method seems (missed).
Sangwhan: API surfce seems fine. Alternative proposal from another chromium person that might change it a bit; seems like improvement, but not that significant a change.
Sangwhan: alternative is what Kenneth linked to: https://github.com/WICG/native-file-system/issues/19#issuecomment-490579093
Lukasz: Questions as well about aspects in security & privacy quesstionnaire. Would be nice if they clarify. They say they encourage UAs to protect access to file based sensors, i.e. special files like /dev/something, but little clear on that. They also say API will behave differently in PWAs. I my view this is probably among the first such approaches, if not the first. Would be reasonable to have a close look at that. We did not receive feedback, bump?
Alice: They also requested github issues in their repo; they might not be tracking comments in our issue.
Hadley: Naive question: why is this useful to take out of the UA and status quo for the way things are done and put into this set of defined APIs? That is, what's the benefit of standardizing this rather than having the UA handle things as done now.
Sangwhan: Not there now.
Kenneth: Today you can get a handle in indexed db. Today you can't build something like Visual Studio code on the web.
Sangwhan: Supposyou wanted build a web app that handled a 100mb file-- can't do incremental updates today.
Alice: Somebody file an issue on their repo to sugest adding those examples directly to their explainer (multi-file editor, large file)
Sangwhan: It's there (multi-file editor), in the goals section.
Alice: large file isn't.
Alice: And seems like the words could be fleshed out a bit.
Hadley: I don't see a use case described in this explainer; it's a bunch of functionality. Not written to say what the problem for the user is.
Lukasz: persistent access to filesystem -- user gives permission for persistent access to directory. Visits site again, no permission prompt. Streamlining access to the file system. Yes, possible Privacy issue. How to mark that this kind of access is actually being retained?
Peter: That you can send a handle through postMessage
also scares me.
Kenneth: only on same origin, right?
Peter: checking
Kenneth: Other point, could only access handles in specific cases. Not really writtten here, but was told that's the idea.
Sangwhan: One question I did have: any way to do equivalent of mmap
-- I think the answer is no.
Kenneth: file an issue?
Sangwhan: So what do we think about this?
Hadley: At a bare minimum would like the explainer to explain use cases more.
Lukasz: mmap
would also access /dev/
filesystem?
Sangwhan: /dev/
filesystem is out of scope... says so in the spec... or maybe the explainer?
Lukasz: I asked them in the this ... not really clarified.
Sangwhan: I think "encouraged" is too weak. I'm also uncomfortable going to a point outside of your home directory -- should be as restrictive as possible.
Lukasz: Not only too weak, but also not being written anywhere. Even that would be scary.
Kenneth: Lukasz, could you file your issue on the repo?
Lukasz: I ??? my question but received no response.
Kenneth: Yes, but you should file it in their issue tracker.
Lukasz: ??? these kinds of answers in their issue tracker.
Sangwhan: I put it in there.
Kenneth: Add "in response to" so we get a link back?
Sangwhan: done
Peter: more to discuss with this issue?
Hadley: I'm opening an issue on their repo about the explainer.
Kenneth: I already did.
David: Are browsers going to be willing to ship this given security risks?
Sangwhan: Should it be PWA-only?
Kenneth: Get a modernized API if not a PWA -- but can't store handles in IndexedDB. How it's going to be implemented in Edge and Chromium.
Kenneth: The other thing, not implementing this all at once in Chrome, staged rollout. Would be nice to know what's in what stages, and what we should be concentrating on.
Sangwhan: I'm also concerned about exclusive access -- multiple tabs operating on the same file at once.
Kenneth: Have ??? about that. They're copying files unless you set specific markers "in place" etc.
Sangwhan: Does "in place" twice reject, lock, etc.?
Kenneth: How does that interact with native apps doing the same?
Sangwhan: Can use OS-level functionality.
Kenneth: file an issue.
Sangwhan: Did you already file it?
Peter: What to do with the issue? Pending feedback at this point? Kick it out a few weeks?
Kenneth: seems to be moving slowly.
Sangwhan: 3 weeks?
Kenneth: TPAC?
Sangwhan: Haven't followed up on issues they've filed themselves. Downprioritized? Vacation?
Peter: 4 weeks then?
Comment by @kenchris Aug 28, 2019 (See Github)
Hi there,
We have been told that the feature will be rolled out in pieces, we would very much like to know the order of this roll out so that we can prioritize the review of this feature. Thanks!
Comment by @mkruisselbrink Aug 28, 2019 (See Github)
We've tried to answer that question in the document linked to from the original review request, as well as in a reply to the question you raised about that in the spec repo.
Anyway, to summarize all that:
In initial Origin Trial (chrome 78), what is currently in the spec with the exception of:
- FileSystemDirectoryHandle.resolve()
- FileSystemWriter.asWritableStream() (or some other way of supporting writable streams)
- Handles won't be Serializable
- FileSystemCreateWriterOptions.inPlace. I.e. all writers will not be in-place writers (or phrased otherwise, all FileSystemWriter instances will have their atomic flag set to true.
Less clear in what order we'll work on these features afterwards, but probably streams and serializabilty will come before inPlace.
Comment by @cynthia Sep 11, 2019 (See Github)
If the rollout is going to happen in multiple stages, it would be extremely helpful for the end users (developers) to get a spec version that only has the parts actually implemented. We see this pattern quite often nowadays, and think it's a point that the platform needs to improve on.
The subset of functionality shipping seems fine to me. (I can't speak for the rest of the TAG here, as this is one of those complicated topics where there are multiple opinions)
Comment by @foolip Nov 20, 2019 (See Github)
I did a review of the spec today and found this linked in an issue, so I thought I'd summarize what seem like the most important existing or new issues to me.
On API shape:
- Why not use ReadableStream/WritableStream/WritableStream*Writer
- Async iterables are happening
- Consider renaming FileSystemDirectoryHandle's getFile() method to getFileHandle()
- Provide append functionality / Let write() resolve with the number of bytes written? (bugs seem likely)
On behavior/interoperability:
- Interoperability of security and privacy considerations
- When is user activation / prompting required?
- How do MIME types map to native file systems?
- Handling of non-Unicode handle names / symlinks
Finally, cloud drive considerations are probably not a big deal, but could affect the shape/behavior of the API given feedback from experts on that.
Comment by @mkruisselbrink Nov 20, 2019 (See Github)
Thank you for the thorough review @foolip. I plan to spend most of next week working on the spec, so should have updates in a week or two.
Comment by @kenchris Dec 4, 2019 (See Github)
A lot of good work seems to be going on here and I also talking F2F with @oyiptong at a few occasions.
As the shipping of this feature will happen in parts, please advice us when you have something ready that you would like review of, so that we can concentrate on those pieces first.
@lknik could you look at this in details from security point of view?
Comment by @mkruisselbrink Feb 3, 2020 (See Github)
Sorry for the delays here, our current plan is to do more work on the spec side of things/make sure it's up to date with what we're implementing over the next couple of weeks, so by the end of February we should have something ready that we'd like a review of.
Comment by @torgo Feb 10, 2020 (See Github)
Just noting that in the MDN Web Developer Needs Assessment survey results, "file system" access was reported as one of the key things developers felt was missing from the web. https://insights.developer.mozilla.org/
Discussed
Feb 17, 2020 (See Github)
Peter: has face-to-face milestone as well
Discussed
Feb 17, 2020 (See Github)
[some discussions on covid-19 and impact on next f2f]
Sangwhan: general binary blob access - a filesystem - is useful. or do developers want access to "the" filesystem - which is a can of worms.
Dan: agreed it's a can of worms - interested to see what this proposed design is...
[bumped to f2f and dan will try to get more info about the mdn survey
Comment by @cynthia Mar 3, 2020 (See Github)
@kenchris and I discussed this during the Wellington F2F. So far, it looks like not much has changes since we last looked at it. (on the Github repo)
so by the end of February we should have something ready that we'd like a review of.
Is there a work-in-progress we can look at? Have there been any significant changes in the design?
Comment by @mkruisselbrink Apr 13, 2020 (See Github)
I finally have been able to make some progress here, once wicg/native-file-system#168 and wicg/native-file-system#169 land there should be some normative spec text for everything in the spec at least (and the integration with writable streams has landed in the spec as well). Having said that, there are still plenty of known open issues and open questions.
Comment by @mkruisselbrink Apr 13, 2020 (See Github)
Some particular questions I would love your input on:
-
different methods vs multiple options for one method (and different return types depending on what options were passed in), WICG/native-file-system#25. Currently we have one method with a bunch of options, and depending on the options passed in you get a different return type (files vs directories, single handle vs sequence of handles). It might be nicer to have separate methods with return types that don't depend on the options, but having it all be one method doesn't seem crazy to me either.
-
And WICG/native-file-system#158, which is basically the question what the best way to integrate with async iterators would be: should a directory handle itself be (async) iterable with its "values" being the files/directories it contains, or should we keep the current API where there is a method that returns an async iterable. I'm leaning towards the first, but it feels a bit awkward to me to have a "values" getter rather than something more appropriately named (I suppose we could have both as well, but that might be the worst of both?)
So definitely would welcome your feedback on those issues (and anything else of course).
I hope to address some more of the earlier filed feedback and outstanding issues in the next week or two as well, again sorry for the delays.
Comment by @kenchris Apr 29, 2020 (See Github)
Generally, the return type should not differ depending on what arguments a method takes or what a user selected. Multiple and single file can be solved by always returning an array and letting that array just have the size of 1 in the case of a single file. I believe the existing input type=file support works that way
Comment by @cynthia May 27, 2020 (See Github)
We've provided feedback on the first question here: https://github.com/WICG/native-file-system/issues/25
It looks like the second question has already been addressed on your end.
Comment by @mkruisselbrink Aug 4, 2020 (See Github)
Yes, thank you. I think the spec is getting pretty close to being something I'm happy with, and we're gearing up for shipping what is currently in the spec (or something very close to that) in hopefully Chrome 86. We've made a number of API changes in response to feedback we received (documented in https://github.com/WICG/native-file-system/blob/master/changes.md), and also resolved a number of spec issues. One more spec-internal, but large change is that we changed how permissions are specified to integrate with the permissions API (in WICG/native-file-system#200).
I would like you're input on WICG/native-file-system#210. Currently the spec defines a number of methods on the global, but another option would be to define those in their own FileSystem namespace. Not sure what the trade-off is between the two options. https://w3ctag.github.io/design-principles/#example-09c8f087 mentions to use namespaces instead of singletons, but doesn't really elaborate on just having new methods on the global vs having those new methods in a namespace. Any guidance you can give here?
I'd also like to point your attention at WICG/native-file-system#192. Our explainer used to say that we weren't going to integrate with drag&drop initially, but we've been having an intern work at that, so we might be including that in the initial version of the API we ship after all.
Other than that, we have a number of spec issues we're still working on resolving, but since these are unlikely to result in breaking API changes we haven't prioritized them so far.
And after shipping we are likely going to keep working on adding new features/methods to the API.
Comment by @cynthia Sep 23, 2020 (See Github)
@kenchris and I discussed this during our "Cork" F2F.
The changelog was immensely helpful! Maybe we should make this a recommended practice. Thanks a lot for taking the time to write that.
Apologies that our feedback took so long. This is moving forward fast and it looks like our feedback wasn't timely - but we retroactively looked at WICG/file-system-access#210, and we don't have a strong opinion on whether this should be namespaced or in global. Since the amount of APIs being added is moderately low, it does seem fine to be on global - unless you have plans to add a lot more related to this capability in the future. (Although we don't quite see what is missing, so it would be good to know if you do have such plans.)
We also looked at the D&D integration, and it looks good to us. Good to see this being added, as we can see this improving user experience.
Since this seems to be in good shape, we are happy to consider this review finished, and would love to see the developers start using this. (We did see some misleading press about this, presumably written by someone who hasn't fully read the spec.) One last point - it looks like the work has reached a point where it needs a home (meaning, a working group) - is that something that is being discussed?
Comment by @cynthia Sep 23, 2020 (See Github)
Group consensus is that this is good to close. Thanks for incorporating the feedback!
Comment by @LeaVerou Mar 15, 2023 (See Github)
Hi there, I know this is closed, but just a quick question wrt API design that I didn't see being brought up during the actual review: Why is this a set of global functions, instead of keyed off some kind of namespace?
Comment by @mkruisselbrink Mar 15, 2023 (See Github)
Hi there, I know this is closed, but just a quick question wrt API design that I didn't see being brought up during the actual review: Why is this a set of global functions, instead of keyed off some kind of namespace?
That was raised by me in https://github.com/w3ctag/design-reviews/issues/390#issuecomment-668755957, and answered by a member of the Tag in the following comment.
Comment by @LeaVerou Mar 15, 2023 (See Github)
Hi there, I know this is closed, but just a quick question wrt API design that I didn't see being brought up during the actual review: Why is this a set of global functions, instead of keyed off some kind of namespace?
That was raised by me in #390 (comment), and answered by a member of the Tag in the following comment.
This was before my time, but IMO there should have been stronger guidance against this. It's not just about namespacing, I think it's more confusing for authors when functions are just added to the global scope like this, as there's nothing tying related parts of an API together. I guess this ship has sailed now, but I'll open a design principles issue so we can avoid it in the future.
OpenedJun 24, 2019
こんにちはTAG!
I'm requesting a TAG review of:
Further details:
You should also know that...
As mentioned above, we fully expect to be iterating on the best shape of this API for at least the duration of the origin trial (i.e. most of 2019). We have some idea of what we want the API to look like, and what use cases we want to support, but also expect to learn from the origin trial that perhaps we were wrong about what is needed for which use cases.
We'd prefer the TAG provide feedback as (please select one):