The sub-millisecond writes with data in S3 is false and impossible. If you look at the benchmark the fsync is not timed, so this is just the latency of either the network or in kernel file operations depending on the mount settings
I hate it when databases celebrate their performance without synchronous flushing. You should be clear about data loss window (which should be zero for committed transactions by default!) and the flushing interval to persistent storage.
I'm okay if you batch writes, I'm okay if you offer a low-latency mode with less durability, but by being unclear about this it just feels like a scam.
Yeah in this case the footnote to the write latency specifically says “at rest in S3”, which is what caused me to go look at the source. To be clear I have no problem with the ZeroFS of only flushing on fsync.
I am very excited for object storage first systems like this to leverage low latency zonal storage for write ahead logs to keep the disaggregated storage but greatly reduce write latency. That ends up being more expensive, but is likely a good tradeoff in lots of cases I have seen
Where is the "data loss window"? Between nodes or between the client and the infra?
The front page lists under Capabilities:
> 4.8
> Honest fsync
> A successful fsync means every acknowledged write is durable in S3. If a failover may have lost unflushed writes, the next fsync returns an error instead of a false success.
Someone else mentioned: "write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable."
---
It sounds like all writes are written synchronously to at least one node but failovers/replicas are just eventually consistent. If so, latency between nodes is not within ZeroFS's control and including. Or are you saying that the latency is impossible for even a single node? If so, that would mean much more than just a footnote is needed.
---
I don't see an issue with the benchmark but I might not be looking in the right place.
ZeroFS aims to be a POSIX filesystem, the semantics here are the standard ones (ext4, xfs behave the same): write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable.
Nothing wrong with that, but you should remove the “at rest in S3” footnote from the write latency on the frontpage of the website, because that is not what is measured
Incidentally the fact that you could read and write from essentially thousands of harddrives at once for even a tiny file makes you wonder what novel use cases you could get out of that.
Read/write operations in object storage are _far more_ expensive than stored bytes. I'm always afraid of anything that abstracts over S3/GCS access specifically for that reason.
Since you are harnessing the sorcery of AI, have it write really good benchmarks, run tests and comparisons on competitive products, (and publish them), look up common pitfalls, often requested features, run security analysis.
Also with marketing texts, write your self first and then you can ask AI to hone it or give you feedback. AI slopped marketing text is visible from miles and really, really puts people off. Even if the product itself would be fine, there is some much slop slushing around in the pipes at the moment.
I really like this project and want to see it succeed! Don't let naysayers wear you down.
Thanks for being explicit, AI written marketing site. Wouldn't have been able to figure that out! Every currently maintained and reasonably popular open source project either runs CI in public or makes the tests extremely easy to run.
> These are asciinema recordings of real terminal sessions, rendered as text rather than video. Playback caps idle pauses at two seconds and changes nothing else.
Thanks? This sounds like it's the LLM's response to the prompter, not something you should display on the page itself...
I see this all the time in code reviews at work. Extremely verbose comments that teach the clueless author how things work but have no place in the final code: aside from codebase not being a coding tutorial, they are also incredibly specific and would become stale and incorrect in matter of weeks.
I feel bad for actually liking that part now. Capping pauses at 2 seconds would show you where it hung 2+ seconds without wasting your time. Smart I thought.
If one of your goals is to get others to adopt the software, I recommend you redo the marketing page and readme from scratch. Delete them without looking at them again, then hand write the content for them. Once you have the content, you call tell an LLM to format it into a nice landing page, but strictly keep your wording without changes.
I prototyped something like this for fun a long time ago. Treating s3 like a bucket of blocks seemed intuitive way build a scalable filesystem. Arguably ceph and luster are doing something similar except with a seperate metadata servers to serve the hotter content.
I think the critical thing you will need to explain is durability and loss window. Making some guarentees on failure modes would go a long way towards making me believe i can run operations on something like this.
With AI you should be able to do some exhaustive testing both for load, power loss, server loss, etc. Anxious to see the potential results
How does this compare to JuiceFS or SeaweedFS in terms of metadata latency? The LSM tree approach is interesting but compaction pauses on a remote-backed store seem like they could be painful.
If you actually bench it locally (local S3, actually writes to disk for "staged" operations), ZeroFS performs horrifically. Ceph blows it out of the water. I don't have the exact numbers, but when I was building a toy CoW distributed block device and filesystem I did a perf matrix, and ZeroFS (even with an hour of codex tuning it) was never within the same 1 or 2 orders of magnitude perf-wise.
Sure, Ceph isn't "S3-backed", but when you're talking about an actual filesystem or block device (the thing that does lots of small-IO), you care more about io-perf than sequential.
Large sequential jobs that everyone is targeting now (ie AI workloads) can use s3 directly just fine, because they don't have decades of code built on top of the filesystem.
There are a couple of different ways of using Ceph with a filesystem: (1) CephFS, or (2) RBD (Ceph block device) volume mounted and then create filesystem on the mounted RBD volume. Historically, the RBD approach would likely have been the more common of the 2. Which of these 2 ways were you referring to?
I believe the first version of this required the metadata to be stored on the ZeroFS server, making HA kinda hard.
This has changed now that if I stop the server and create a new instance with the same configuration file it'll pickup the existing metadata from the bucket?
We have done loads of research into using object storage wherever we can (given how cheap it is compared to SSDs), and so far it seems like making your application object store-aware is a far surer bet than abstracting S3 behind the file system. The behavior is just too different.
I'm more interested in applications that cleverly use object storage, e.g. AutoMQ, which is quite compatible with Kafka APIs but needs no HDDs.
The page doesn’t load anything for me… I block JS by default, & something that should be informational is hiding it’s content behind scripts for some reason.
NFSv4 is a hard beast to implement correctly, with a lot of protocol surface (state, compound ops, delegations) for benefits ZeroFS mostly gets through 9P with extensions, over a much simpler protocol: https://www.zerofs.net/docs/9p-extensions. NFSv3 stayed in ZeroFS mostly for client compatibility.
The 128 KiB chunk size is an interesting tradeoff point — small enough to avoid wasting bandwidth on partial reads, but you're still paying per-request overhead on S3 (both cost and latency) for anything that reads across many chunks. Curious how ZeroFS handles read-ahead/prefetching for sequential access patterns, since that's usually where these abstractions either save you or quietly rack up request costs. Tools like JuiceFS and SeaweedFS handle this differently (local metadata cache + larger block coalescing) — would be interesting to see a head-to-head on request volume for the same workload.
The sub-millisecond writes with data in S3 is false and impossible. If you look at the benchmark the fsync is not timed, so this is just the latency of either the network or in kernel file operations depending on the mount settings
I hate it when databases celebrate their performance without synchronous flushing. You should be clear about data loss window (which should be zero for committed transactions by default!) and the flushing interval to persistent storage.
I'm okay if you batch writes, I'm okay if you offer a low-latency mode with less durability, but by being unclear about this it just feels like a scam.
Yeah in this case the footnote to the write latency specifically says “at rest in S3”, which is what caused me to go look at the source. To be clear I have no problem with the ZeroFS of only flushing on fsync.
I am very excited for object storage first systems like this to leverage low latency zonal storage for write ahead logs to keep the disaggregated storage but greatly reduce write latency. That ends up being more expensive, but is likely a good tradeoff in lots of cases I have seen
Where is the "data loss window"? Between nodes or between the client and the infra?
The front page lists under Capabilities:
> 4.8 > Honest fsync > A successful fsync means every acknowledged write is durable in S3. If a failover may have lost unflushed writes, the next fsync returns an error instead of a false success.
Someone else mentioned: "write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable."
---
It sounds like all writes are written synchronously to at least one node but failovers/replicas are just eventually consistent. If so, latency between nodes is not within ZeroFS's control and including. Or are you saying that the latency is impossible for even a single node? If so, that would mean much more than just a footnote is needed.
---
I don't see an issue with the benchmark but I might not be looking in the right place.
The [sequential writes](https://github.com/Barre/ZeroFS/blob/ec32199d48d0409d4cccd44...) and [append-only writes](https://github.com/Barre/ZeroFS/blob/ec32199d48d0409d4cccd44...) start where I'd expect and `success: true,` should equate to `fsync()` as previously mentioned.
(Apologies if I got this wrong, still learning this)
ZeroFS aims to be a POSIX filesystem, the semantics here are the standard ones (ext4, xfs behave the same): write() is buffered (that's the batching) and "committed" maps to fsync(), which returns only once data is durable.
Nothing wrong with that, but you should remove the “at rest in S3” footnote from the write latency on the frontpage of the website, because that is not what is measured
Incidentally the fact that you could read and write from essentially thousands of harddrives at once for even a tiny file makes you wonder what novel use cases you could get out of that.
From the docs:
> ZeroFS fetches object data in 128 KiB parts
Read/write operations in object storage are _far more_ expensive than stored bytes. I'm always afraid of anything that abstracts over S3/GCS access specifically for that reason.
One of the reasons why ZeroFS seems interesting is they use SlateDB under the hood, which optimizes the requests that hit S3 behind the scenes.
Especially that the “one fetch” is who knows how many reads and retries under the hood.
I’ve seen things like this before; your key differentiator needs to be efficiency and safety compared to other options.
OP this is the best advice here.
Since you are harnessing the sorcery of AI, have it write really good benchmarks, run tests and comparisons on competitive products, (and publish them), look up common pitfalls, often requested features, run security analysis.
Also with marketing texts, write your self first and then you can ask AI to hone it or give you feedback. AI slopped marketing text is visible from miles and really, really puts people off. Even if the product itself would be fine, there is some much slop slushing around in the pipes at the moment.
I really like this project and want to see it succeed! Don't let naysayers wear you down.
Thank you, appreciated!
> The test suites run in public CI.
> Each card links to the CI pipeline.
Thanks for being explicit, AI written marketing site. Wouldn't have been able to figure that out! Every currently maintained and reasonably popular open source project either runs CI in public or makes the tests extremely easy to run.
I got the same vibe from
> These are asciinema recordings of real terminal sessions, rendered as text rather than video. Playback caps idle pauses at two seconds and changes nothing else.
Thanks? This sounds like it's the LLM's response to the prompter, not something you should display on the page itself...
I see this all the time in code reviews at work. Extremely verbose comments that teach the clueless author how things work but have no place in the final code: aside from codebase not being a coding tutorial, they are also incredibly specific and would become stale and incorrect in matter of weeks.
I feel bad for actually liking that part now. Capping pauses at 2 seconds would show you where it hung 2+ seconds without wasting your time. Smart I thought.
Thank you for the feedback, the idea behind this was to say "We make claims that are backed by workflows you can verify". I'll improve the phrasing.
Entrusting data storage to a vibe coded filesystem seems imprudent.
Is it? :)
Ask me anything!
Here is my unsolicited advice:
If one of your goals is to get others to adopt the software, I recommend you redo the marketing page and readme from scratch. Delete them without looking at them again, then hand write the content for them. Once you have the content, you call tell an LLM to format it into a nice landing page, but strictly keep your wording without changes.
That's fair advice, thanks.
FYI it looks like some of your comments are getting auto-flagged by the HN moderation system and marked as dead
I prototyped something like this for fun a long time ago. Treating s3 like a bucket of blocks seemed intuitive way build a scalable filesystem. Arguably ceph and luster are doing something similar except with a seperate metadata servers to serve the hotter content.
I think the critical thing you will need to explain is durability and loss window. Making some guarentees on failure modes would go a long way towards making me believe i can run operations on something like this.
With AI you should be able to do some exhaustive testing both for load, power loss, server loss, etc. Anxious to see the potential results
previous Show HNs
- https://news.ycombinator.com/item?id=48496242 -- 11 points | 20 days ago | 7 comments
- https://news.ycombinator.com/item?id=45174724 -- 64 points | 9 months ago | 40 comments
blog post 3 days ago: https://news.ycombinator.com/item?id=48712122
How does this compare to JuiceFS or SeaweedFS in terms of metadata latency? The LSM tree approach is interesting but compaction pauses on a remote-backed store seem like they could be painful.
If you actually bench it locally (local S3, actually writes to disk for "staged" operations), ZeroFS performs horrifically. Ceph blows it out of the water. I don't have the exact numbers, but when I was building a toy CoW distributed block device and filesystem I did a perf matrix, and ZeroFS (even with an hour of codex tuning it) was never within the same 1 or 2 orders of magnitude perf-wise.
Sure, Ceph isn't "S3-backed", but when you're talking about an actual filesystem or block device (the thing that does lots of small-IO), you care more about io-perf than sequential.
Large sequential jobs that everyone is targeting now (ie AI workloads) can use s3 directly just fine, because they don't have decades of code built on top of the filesystem.
There are a couple of different ways of using Ceph with a filesystem: (1) CephFS, or (2) RBD (Ceph block device) volume mounted and then create filesystem on the mounted RBD volume. Historically, the RBD approach would likely have been the more common of the 2. Which of these 2 ways were you referring to?
I believe the first version of this required the metadata to be stored on the ZeroFS server, making HA kinda hard.
This has changed now that if I stop the server and create a new instance with the same configuration file it'll pickup the existing metadata from the bucket?
> I believe the first version of this required the metadata to be stored on the ZeroFS server, making HA kinda hard.
Metadata has always been in the bucket itself.
For HA, there's now a "replicated mode" if you want automatic failover:
https://www.zerofs.net/docs/high-availability
The HA certainly makes this more of an alternative to JuiceFS now.
Would need something like HAProxy to correctly handle failover for NFS though?
See also: JuiceFS, S3FS, and quite a few others.
We have done loads of research into using object storage wherever we can (given how cheap it is compared to SSDs), and so far it seems like making your application object store-aware is a far surer bet than abstracting S3 behind the file system. The behavior is just too different.
I'm more interested in applications that cleverly use object storage, e.g. AutoMQ, which is quite compatible with Kafka APIs but needs no HDDs.
s3fs doesn't provide posix semantics. It's good enough™ for some uses, but not comparable to what this one is ostensibly providing.
I agree that most usecases are best suited just using S3 directly, but I would check this out if you want an S3 based filesystem: https://fiberfs.io/
Worth mentioning FiberFS: https://fiberfs.io/
I believe it just recently launched.
I’m more interested in a generic (offline) caching layer for example SMB or NFS. Preferably with doing conflict resolution like Dropbox
Seems purely vibecoded
wonder when we get agents good enough that we can't say vibecode any more and have to say 'code'.
there was slop with ai jesus but now gpt image is just a photo with hidden watermark
Why does this landing page load js from merklemap.com?
The page doesn’t load anything for me… I block JS by default, & something that should be informational is hiding it’s content behind scripts for some reason.
Both projects have the same author.
Just a self hosted plausible instance :)
There is some kind of advertising push for this going on. I got Twitter ads for this repo.
I don’t get why they went for NFSv3, v4 is quite old and I can’t think of any reason why you would choose v3 over v4.
Because it’s harder. It’s a stateful protocol and has a lot more features
NFSv4 is a hard beast to implement correctly, with a lot of protocol surface (state, compound ops, delegations) for benefits ZeroFS mostly gets through 9P with extensions, over a much simpler protocol: https://www.zerofs.net/docs/9p-extensions. NFSv3 stayed in ZeroFS mostly for client compatibility.
Thanks for explaining it.
That name makes it sound like your files end up in /dev/null.
So, can I run this on top of RustFS? And RustFS on top of this?
Kind-of should work :)
Under the hood, S3's storage nodes are also built on a log-structured file system: https://cdn.amazon.science/77/5e/4a7c238f4ce890efdc325df8326...
(Not posix compliant because it doesn't need to be.)
The 128 KiB chunk size is an interesting tradeoff point — small enough to avoid wasting bandwidth on partial reads, but you're still paying per-request overhead on S3 (both cost and latency) for anything that reads across many chunks. Curious how ZeroFS handles read-ahead/prefetching for sequential access patterns, since that's usually where these abstractions either save you or quietly rack up request costs. Tools like JuiceFS and SeaweedFS handle this differently (local metadata cache + larger block coalescing) — would be interesting to see a head-to-head on request volume for the same workload.