#566: Review of NativeIO
Discussions
2020-11-23
Ken: if you want to implement a db in webassembly - implementing it on top of indexdb not good for performance. Alternative might be to use filesystem access API. You kind of want a big blob of data to read from and write to. Filesystem API has a lot of permissions issues... This is a different API - similar in shape to filesystem access but no bloat - because security is different. Can it be abused? Very nice place with file access - it's mutable.
Dan: left comment asking for implementer info and more info on use cases...
Ken: need to look closer at this.
[disvcussion of this vs. filesystem access]
Ken: video editing use case... other cases where performance is required..
Dan: we're back to primitives vs not-primitives
Ken: a native app will use a bit of the hard drive...
Ken: I don't think it will work for a db like mysql because the binary file for the webassembly will just be too big.
Ken: ... other cases like video editing or machine learning ...
2020-11-30
Ken: seems to be orign-bound filesystem. Different to filesystem - you can open directory. Different sites can use the same directory.
Dan: So it might better be called "orgin bound virtual filesystem"... the current name is quite scary.
Ken: of course with this they don't need the same kinds of dialogs. I'm still worried about quota. Video editing, etc... needs lots of disk space. How is that managed? Handled with usual quota things.
Dan: left a comment
Sangwhan: let's imaging a scenerio where you use something like this to implemnt a PCB CAD webapp - you'd have multiple files... if you bind it to an origin you can't take it to another tool...
Dan: you'd need to export.
Ken: if you want to bring a video to another tool to say add special effects.
Sangwhan: moving between different CAD applications is a pretty common case. The filesystem API does do this.
Dan: trade-off between privacy ....
Sangwhan: what if you want to share between native app and this?
Ken: you'd need to use filesystem API
Sangwhan: these APIs don't acknowledge eachother...
Ken: yes could it be an option on the other API? Give me sandboxed filesystem?
Sangwhan: yes it's a sandboxed filesytem where you can't access the files... I don't see why you would use this over IndexDB... Except for porting legacy application...
Ken: music editing, video editing....
Sangwhan: yeah but this design doesn't make sense for music editing... You have to bring in small pieces of video / audio, assets...
Ken: You could use this in combination with filesystem access... or you get them from a cloud system...
Yves: could you add a few things to make Cache API use Native API... immutable flag... persistence... store until this freshness expires...
Sangwhan: something you can mark as ephemeral...
Yves: it would be good if instead of creating a new system, create a lower-level system that you could build something on top of...
Dan: ... and what about storage buckets?
2021-01-11
Dan: Previously left a bunch of feedback saying given the characteristics of what ??? a file, how can I share this file with my web application... how can these files be accessed from the OS, how does it relate to storage buckets, should there be ephemeral mode... and what's the venue
... They responded saying it should be in WebApps...
Sangwhan: did we ever ask about fsync? Flush to disk. I don't remember seeing that, seeing any guarantees.
Ken: This is a performance thing ...
Sangwhan: There are times when you would definitely want to flush.
... say you put a flush on close... (missed)
Ken: Talking about having a delete on close, could have a flush on close.
Sangwhan: If execution context is destroyed... want something like Beacon?
Ken: There is a flush function actually, looking at the API..
Sangwhan: Doesn't come with guarantees
Ken: (quoting from explaiener)
Sangwhan: Reliable as long as flush finishes before execution context is destroyed.
Ken: It's a good question.
Sangwhan: A lot of things don't guarantee anything once you lose the execution context. All bets are off.
Dan: Same problem we've encountered elsewhere, API being introduced into the platform, a capability that is overlapping with similar capabilities with some significant differences, being introduced as a completely different technology that creates bloat.
Ken: I left feedback to that effect... for a start the name is important, they have decided to change the name to highlight the main use cases.
Dan: It's still using its own concept of a filesystem.
Ken: ... bloat in the implementation ... stay secure... this is for e.g. BYO database, like doing MySQL from WebAssembly. They def know about the filesystem access API, but you can't base one on the other for performance reasons...
Dan: That's an implementation detail; from a dev perspective feels like it should be the same thing, I want to open a file, how do I know which one to use? UAs should be abstracting that implementation difference away, making the dev experience more straightforward
Sangwhan: We have the abstractions, but not the unabstracted version, which this provides. This is the lower level.
Dan: So you think they've adequately addressed our questions?
Ken: Haven't finished designing yet so hard to say, but they're aware of the feedback..
Sangwhan: Not seeing solutions yet to the problems we raised
Ken: But they're still working on it.
Dan: If we've left the feedback we want to leave and they've taken it on, we should probably close with some directions... consider the dev experience of filesystem-like APIs overall when coming up with their final design
Ken: Try to keep the APIs as similar as possible... happy with (?), happy with the name - doesn't sound like how I access a file...
... not access to local files, more like a cache kind of thing...
... you don't have those files, you can't access them from the OS in general.
Sangwhan: This is not for "common" developers, this is for not having to reimplement the whole storage part when you're porting an app to webassembly.
Ken: Or ... people could use the FS access API and emulate this, but then an external application could modify those files and potentially even trigger browser bugs taht way.
Dan: Still feels like why isn't this an optional feature... or an option on FS access API?
Ken: It's not a filesystem
Sangwhan: If you get a handle from the FS API you might be able to ??? these APIs, but it's nto directly related.
Ken: It's like a virtual filesystem.
Sangwhan: This is like libc level.
Dan: Think we should close this and leave the feedback we've left. Propose close, close in plenary. Should be encouraging them to consider the complexity they're adding, thinking about ways to simplify.
Sangwhan: Would like folks to consider the use cases and concerns that we've noted above.
... Oh, and a flush guarantee.
Ken: I think I'm ok with the name. It doesn't say filesystem.
Dan: We've talked about how this is origin-bound and that's the web security model, but also asked about access across origins... kind of conflicting feedback. Intention was just to ask if there's a use case for cross-origin access. If there were that access, it would be problematic... unless we're talking about a first party sets scenario.
Ken: I assume it would have to be... don't want to transfer my database to another site
Sangwhan: You've got CAD, or Eagle... generating PCBs... want to be able to transfer that data from one app to another. Audio editor to video editor.
Ken: Write it to a file?
Sangwhan: It's possible by binding the FS access API... if a FS access API has these handles you can flush a file to disk, then load it up from the other app.
Ken: This is a pretty big use case... a lot of native apps today are looking at doing this kind of thing.
Sangwhan: It's not an imaginary use case. If these files are trapped in virtual FS land, it's inaccessible to the user. You can make SQL Lite easier to use, but that's just one use case...
Ken: Would you use this API for this instead of the FS Access API? When would I use this API instead.
Dan: leaves comment on developer complexity
Sangwhan: Legacy CAD format that's implemented in C, for example... don't want to port it all over to JS...
Ken: If we can write to a non-virtual file, that would solve this concern...
Sangwhan: If there were a connection, yes. That connection is currently missing.
... Will try and leave a comment around this.
Ken: Writing it to a different file mitigates Dan's concern
Sangwhan: Yes
Ken: It's like saving a file.. ... delegating access to storage from one origin to another origin... that's a can of worms.
OpenedOct 28, 2020
HIQaH! QaH! TAG!
I'm requesting a TAG review of NativeIO.
NativeIO is a storage API that resembles a very basic filesystem, with direct access to stored data through buffers and offsets. Our goal is to give developers flexibility by providing generic, simple, and performant primitives upon which they can build higher-level components. It's particularly well suited for Wasm-based libraries and applications that want to use custom storage algorithms to fine-tune execution speed and memory usage.
Further details:
We'd prefer the TAG provide feedback as:
🐛 open issues in our GitHub repo for each point of feedback