#772: Sync Methods for FileSystemSyncAccessHandle in File System Access API
Discussions
Discussed
Oct 10, 2022 (See Github)
bump plenary
Comment by @littledan Oct 27, 2022 (See Github)
I'm wondering, what's the motivation for the sync API? I think async APIs should be friendly to WebAssembly in the near future, given that WebAssembly JavaScript Promise Integration reached Phase 3 in the Wasm CG.
Comment by @a-sully Oct 27, 2022 (See Github)
There's a lot of great discussion by WASM developers on https://github.com/whatwg/fs/issues/7 which illustrate the need for this change. This comment (sampled below) provides a pretty good summary of my thoughts:
From my perspective as a Chromium engineer, we are currently focused on making SyncAccessHandles as performant as possible to support C(++) applications being ported to the web via WASM, but we are very interested to hear developer feedback about whether an asynchronous alternative to SyncAccessHandles is worth pursuing. One could imagine an async alternative could be more webby: available from Window contexts etc, using a streams-based interface with built-in queueing, etc. To this point we haven't seen much developer demand for this async alternative. If you have compelling use cases for an async alternative, we'd love to hear your feedback :)
Okay, so here's the reality of the current landscape:
- asynchronous WASM is not performant enough to support the most powerful web applications (for now!)
- applications porting C(++) code to WASM expect a synchronous, Posix-like file API
- a split sync/async interface is a no-win compromise
I'm excitedly following the developments to improve async support in WASM. My hope is that this will allow an async alternative to SyncAccessHandles to have comparable performance characteristics, at which point any web developer looking for fast storage will have no reason to choose the more restrictive SyncAccessHandles.
That being said, it's also unclear to me whether C(++) applications being ported to WASM would be able to use or benefit from this async alternative as easily as SyncAccessHandles. There may always be a need for a synchronous file API to support porting these applications to the web.
The current split interface is the worst of both worlds. We'll never be able to expose SyncAccessHandles to anything other than DedicatedWorkers due to the existence of sync methods. However, the split sync/async interface forces the application to write effectively sync code anyways, since you can't queue operations due to the sync methods. The result is a significantly slower and more complicated interface than if all the methods were sync. I'd like to see SyncAccessHandles be the MOST useful for WASM-compiled applications, while keeping an ear out for WASM ecosystem async support + developer demand for an async alternative.
Comment by @littledan Oct 27, 2022 (See Github)
I am having trouble following this thread. It sounds like the user-facing requirement is that some C++ code can make a blocking call. My understanding is that this is what WebAssembly JS Promise integration provides. These are two ways to implement the same high-level user requirement; it's not the kind of thing user demand will inform one way or the other.
Is the concern that JSPI is lower performance? Do you have any benchmarks that show this penalty? I wonder if this benchmark can be used to fix/further optimize JSPI (whether in its core design or various implementations of it), regardless of in what form this feature ships.
The reason I am interested in this area is, I worked on the design of JavaScript WeakRefs, where we also had a sync method created for Wasm, and removed it for the reasons @fgmccabe mentions in the thread. I am wondering how we should handle similar cases going forward (as this question can reasonably come up for many async APIs, hence it being a relevant concern for the TAG).
Comment by @a-sully Oct 28, 2022 (See Github)
If the high-level concern here is that we don't want to end up in a world with sync and async versions of every API, then, well, I agree :) This API is a particularly performance-oriented exception, not the rule.
The purpose of this API is performance. SyncAccessHandles are intended to provide the most low-level persistent file primitive to the web. This primitive allows sites to bring their own database to the web via Wasm in a way that's orders of magnitude faster than IndexedDB, for example.
Also I realize this isn't clear based on the issue description, but SyncAccessHandles are only available from DedicatedWorkers specifically because we don't want to impact the responsiveness of sites.
Discussed
Nov 28, 2022 (See Github)
Amy: Sync and async discussion... in the thread...
Sangwhan: I can comment on this as well... In the web NN api... ...Interesting work-around... sync APIs only available on workers... to mitigate the damage. We could incorporate that guidance into the design principles...? In the context of NN API that's fine.. The sync suffix guidelines maybe applies here as well. We do have to have a deprecation path for these things as well. Going to be difficult to remove later on.
Peter: it's not horrible to have sync stuff that's only on workers... possible to feature-detect on the main thread though?
Sangwhan: this would be implementation-specific... On Firefox and Chrome it's possible... We could solicit some feedback - worker-only enforceable?
Amy: question from littleDan in this thread... wondering how we should handle cases going forward...
Sangwhan: he was suggeting we could leave room for WASM promise integration... but that's not shipping yet. So we should wait for that until we make it a principle. Can loop him in to the design principles issues.
Peter: one concern: are we going down the path of adding a full posix layer to browsers?
Sangwhan: probably yes...
Peter: might be worth taking a step back and asking the question - is there a line that we need to draw?
Sangwhan: the posix model doesn't fit with the main web... the threading model doesn't quite work... main()
loop paradigm not compatible with the web... This happens because we want to re-use code on the web, but a lot of it isn't compatible.
Peter: it leads down the path that browsers ship a complete copy of linux...
Sangwhan: there are companies that want to do this.
[Agreed that we should get into this issue, separately.]
Dan: we continue to bang on about developer compexity... this falls into that catetory. What are we trying to do here?
Hadley: What do we need to do on this?
Sangwhan: I'll dig up ongoing relevant work and we can carve out a separate discussion for that, as an architectural thing.
Also a design principle needs to be drafted and reviewed.
Comment by @hcldan Nov 30, 2022 (See Github)
This broke our shipped code. We were using an API that landed in a release of chrome. There needs to be a better way to handle things like this.
We have a POSIX filesystem built on top of these APIs (we do not have the problems of asyncify, not using it) that we've shipped to customers, we were relying on the APIs that landed in a release of chrome (after a long time of being roped off, mind you) and this change broke our customers. We now have to release an emergency fix. Please, PLEASE, find other ways to do this in the future.
Comment by @tomayac Nov 30, 2022 (See Github)
Hi @hcldan! Really sorry to hear that. We did have a blog post aimed at developers that described the breaking change: https://developer.chrome.com/blog/sync-methods-for-accesshandles/. I recommend you subscribe to the blog’s RSS feed if you haven’t already. Again, sorry this broke your product. If there’s better ways we could have reached developers like you or if there are places you looked at but that didn’t hint at the upcoming breakage, please let me know. Ideally in private, as to not spam this thread with unrelated stuff. Thanks!
Discussed
Dec 19, 2022 (See Github)
bumped to next year
Discussed
Jan 9, 2023 (See Github)
bumped to C
Discussed
May 22, 2023 (See Github)
Tess: they want to change it from async to sync because of wasm... "async calls not fully supported on wasm yet"... so ... that yet feels important to me. This is trend we see: Sometimes people want a new feature because some other feature isn't quite right... so this is a stop gap... right way is to fix the other thing... Isn't the right thing for WASM to make async calls work better?
Peter: in general I agree with you - but also, there has been lots of cases of people creating a stopgap which needs to be supported forever... sometimes the stop gap takes longer than fixing the thing they are working around.
Tess: that said, if this is an API to use from WASM and workers then sync might be the right shape for it... If they are design it only from workers or work like context ... then it might be more ergonimic... broadly share the concern with adding sync apis but maybe it's OK when you can't block the main thread?
Peter: I remember discusson on this - part of this is to make more of a POSIX like layer... seems like we're trying to ad POSIX layer across the web platform...
Tess: e.g. cross-compiling C code... if you can make porting to the web easier then maybe it's a good thing?
Peter: I raised a concern: is it a good thing to turn the web into a POSIX layer?
Tess: they did address my issue in a comment here... LittlDan commended in october that async should be available in WASM in the near future... Does WASM now have adaquate support? WASM Promises Integration is at Phase 3.
Tess:
<blockquote> What's the current state here? Given that the main motivation for this is WASM and the WASM people are saying they are working on it from their end, is it still necessary to have this feature? </blockquote>Tess: leaves comment
Comment by @hober May 22, 2023 (See Github)
What's the current state here? Given that the main motivation for this is WASM and the WASM people are saying they are working on it from their end (WASM Promises Integration is at Phase 3), is it still necessary to have this feature?
Comment by @a-sully May 22, 2023 (See Github)
The fully-synchronous interface* is shipped and in active use on all major browsers. Given the usage, I don't think deprecating this feature if feasible, if that's what you're asking.
You bring up a good point, though. Given the progress we're seeing with JSPI would we create the FileSystemSyncAccessHandle
interface at all if we had to do it over again?... I'm not sure. The interface is commonly implemented as a very thin wrapper around a file descriptor. The sync-ness of the JavaScript API continues down the stack - once you hit C++ it's sync calls all the way down**.
I'm not sure what the performance delta is between JSPI and Asyncify, but presumably the basically-no-overhead approach of FileSystemSyncAccessHandle
is faster than what JSPI could possibly be. Which suggests that a sync interface has merits even in a world with JSPI :)
That being said, JSPI makes an async alternative to FileSystemSyncAccessHandle
(which could be used from the main thread) much more attractive. WASM folks have already expressed interest in this: https://github.com/whatwg/fs/issues/41#issuecomment-1397685619
*note that this specific issue only tracks making the entire FileSystemSyncAccessHandle
interface sync. It was initially shipped with a split sync/async interface, which, as I described, in https://github.com/whatwg/fs/issues/7#issuecomment-1201811600, was a "worst of both worlds" situation. By the time this issue was filed it was less a question of "should this sync API exist" but "how can we fix what we've already shipped"
**until you hit the device driver, at least. Though arguably the decision for browsers to implement the FileSystemSyncAccessHandle
interface using synchronous file IO is just an implementation detail, since browsers could choose to use async IO under the hood
Discussed
Jul 3, 2023 (See Github)
Peter: we questioned the utility, they responsed with this already shipped in all major browsers.
Peter: not sure it's actively harmful. It's a foot gun but... there are async alternatives...
Dan: close as overtaken?
Amy: "satisfied with concerns" and keep an eye for deprecation in the future?
Peter: this is part of the whole "do we want to add an entire posix layer to the web platform"...
**we agree to close it with overtaken **
Rossen: why not "unsatisfied"
Peter: not harmful. They asked for tag review then it shipped.
Comment by @plinss Jul 3, 2023 (See Github)
Given that this shipped we're closing the review. While we see the use cases, we do have concerns about adding more synchronous APIs where async may have been more appropriate. We hope that once WASM Promises Integration is available that there is some thought given to deprecating these methods.
OpenedSep 9, 2022
Wotcher TAG!
I'm requesting a TAG review of Sync Methods for FileSystemSyncAccessHandle in File System Access API.
FileSystemSyncAccessHandle
is a file primitive that provides performant access to local files. One of its main use cases is applications porting C/C++ code to Wasm; however, asynchronous calls are not fully supported on Wasm yet, and using Asyncify library as an alternative has substantially degraded performance. Also, a split async/sync interface is not ergonomic, as Wasm-based applications expect a synchronous, POSIX-like file API.Therefore, the proposal is to update asynchronous
getSize()
,truncate()
,flush()
andclose()
methods inFileSystemSyncAccessHandle
in File System Access API to synchronous method, in line with synchronousread()
andwrite()
methods.This change can potentially cause breakage if async methods were used the Promise .then()/.catch()/.finally() methods directly. (If used with
await
, there is no breakage). However, the current usage ofFileSystemSyncAccessHandle
is very low (e.g. zero usage queried by HttpArchive, recent shipping on Chrome-only); therefore, the scope and the impact of breakage is expected to be very minimal. Also,FileSystemSyncAccessHandle
was shipped recently on Chrome M102 and is in the process of being implemented in Firefox, so it is preferred to make this change as soon as possible before the API gets used more widely.Further details:
You should also know that...
FileSystemSyncAccessHandle
is available in Worker-only, thereby all synchronous calls are invoked from Workers, not from window.FileSystemSyncAccessHandle
, and separately, we plan on supporting an async access handle, which may be more appropriate for common use cases, following asynchronous design patterns.We'd prefer the TAG provide feedback as (please delete all but the desired option): đź’¬ leave review feedback as a comment in this issue and @-notify [github usernames] https://github.com/whatwg/fs/issues/7 @dslee414, @a-sully