pierrebarre
9 hours ago
Author here. Thanks for posting this!
It’s been quite a ride building ZeroFS, and I’m happy to answer any questions.
tandr
8 hours ago
I'm thinking about using this and have a few questions:
1. How are hardlinks and duplicate files (same content, different paths) handled?
2. Does deduplication work on a block/chunk level for partially matching files, or does it only look at whole files?
3. Is there any specific integration or handling for Copy-on-Write (CoW)?
pierrebarre
an hour ago
Hardlinks work as you’d expect: multiple paths point to the same inode and data, so a write through one path is visible through the others. Two separate files with the same contents are stored separately.
There’s no deduplication, either whole-file or block-level. That’s intentional, mostly because of the impact it would have on locality.
If by CoW you mean reflinks, those aren’t currently planned either. They avoid the content matching part of deduplication, but still require sharing extents between files and come with similar locality and complexity tradeoffs. Internally ZeroFS is copy-on-write, with immutable segments and checkpoints, but that isn’t exposed as reflinks.
the_duke
6 hours ago
I was investigating the design a little. Two big questions:
A) You notably don't write a recovery log (WAL/journal) for things not yet flushed, so data can be lost. Do you have plans to add this? I think it would be pretty crucial.
B) the system is single writer. Do you have plans for adding horizontal scalability so a writer can be dynamically selected and routed to, transparent to the client? (Or with client cooperation, but without forcing sharding on the user)
chillfox
9 hours ago
Thanks for building this, I am just about to give it ago with my self-hosted Garage cluster.
Does running `stat` against a file require pulling the whole file from s3, or can that be handled by the metadata?
Do you know what backup performance is like for something like borg/borgmatic or restic, especially on follow up runs where most files are just checked.
Is there any particular Redis/Valkey config you recommend when using it for `conditional_put`, or just default config?
Is there any chance for NFSv4 support?
pierrebarre
8 hours ago
Thanks!
stat doesn’t pull the file contents from S3; it only accesses the metadata tree, which is usually cached.
I haven’t benchmarked Borg or Restic specifically. Sequential writes can comfortably reach several Gbit/s. For follow-up runs, if they only stat unchanged files, that should stay entirely in metadata.
The default Redis/Valkey configuration should work fine for conditional_put. NFSv4 is unlikely for now. It would add a lot of surface area, and I’m pretty happy with where the 9P extensions are today.
nubg
5 hours ago
This blog post seems to have been written by an AI based on a prompt you gave it? Can you confirm or deny this?